Horton
Menu

Thread

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Adjust cross-version upgrade tests for seg_out() fix

  2. Rationalize error comments in partition split/merge tests

  3. Add fast path for foreign key constraint checks

  4. Fix assorted pretty-trivial memory leaks in the backend.

  5. Add temporal FOREIGN KEY contraints

  6. Add trailing commas to enum definitions

  7. Remove obsolete executor cleanup code

  1. Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-06-25T12:05:09Z

    Attached is a PoC patch to implement "Row pattern recognition" (RPR)
in SQL:2016 (I know SQL:2023 is already out, but I don't have access
to it). Actually SQL:2016 defines RPR in two places[1]:

    Feature R010, “Row pattern recognition: FROM clause”
    Feature R020, “Row pattern recognition: WINDOW clause”

The patch includes partial support for R020 part.

- What is RPR?

RPR provides a way to search series of data using regular expression
patterns. Suppose you have a stock database.

CREATE TABLE stock (
       company TEXT,
       tdate DATE,
       price BIGINT);

You want to find a "V-shaped" pattern: i.e. price goes down for 1 or
more days, then goes up for 1 or more days. If we try to implement the
query in PostgreSQL, it could be quite complex and inefficient.

RPR provides convenient way to implement the query.

SELECT company, tdate, price, rpr(price) OVER w FROM stock
 WINDOW w AS (
 PARTITION BY company
 ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
 PATTERN (START DOWN+ UP+)
 DEFINE
  START AS TRUE,
  UP AS price > PREV(price),
  DOWN AS price < PREV(price)
);

"PATTERN" and "DEFINE" are the key clauses of RPR. DEFINE defines 3
"row pattern variables" namely START, UP and DOWN. They are associated
with logical expressions namely "TRUE", "price > PREV(price)", and
"price < PREV(price)". Note that "PREV" function returns price column
in the previous row. So, UP is true if price is higher than previous
day. On the other hand, DOWN is true if price is lower than previous
day.  PATTERN uses the row pattern variables to create a necessary
pattern.  In this case, the first row is always match because START is
always true, and second or more rows match with "UP" ('+' is a regular
expression representing one or more), and subsequent rows match with
"DOWN".

Here is the sample output.

 company  |   tdate    | price | rpr  
----------+------------+-------+------
 company1 | 2023-07-01 |   100 |     
 company1 | 2023-07-02 |   200 |  200 -- 200->150->140->150
 company1 | 2023-07-03 |   150 |  150 -- 150->140->150
 company1 | 2023-07-04 |   140 |     
 company1 | 2023-07-05 |   150 |  150 -- 150->90->110->130
 company1 | 2023-07-06 |    90 |     
 company1 | 2023-07-07 |   110 |     
 company1 | 2023-07-08 |   130 |     
 company1 | 2023-07-09 |   120 |     
 company1 | 2023-07-10 |   130 |     

rpr shows the first row if all the patterns are satisfied. In the
example above 200, 150, 150 are the cases.  Other rows are shown as
NULL. For example, on 2023-07-02 price starts with 200, then goes down
to 150 then 140 but goes up 150 next day.

As far as I know, only Oracle implements RPR (only R010. R020 is not
implemented) among OSS/commercial RDBMSs. There are a few DWH software
having RPR. According to [2] they are Snowflake and MS Stream
Analytics. It seems Trino is another one[3].

- Note about the patch

The current implementation is not only a subset of the standard, but
is different from it in some places. The example above is written as
follows according to the standard:

SELECT company, tdate, startprice OVER w FROM stock
 WINDOW w AS (
 PARTITION BY company
 MEASURES
  START.price AS startprice
 ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
 PATTERN (START DOWN+ UP+)
 DEFINE
  START AS TRUE,
  UP AS UP.price > PREV(UP.price),
  DOWN AS DOWN.price < PREV(DOWN.price)
);

Notice that rpr(price) is written as START.price and startprice in the
standard. MEASURES defines variable names used in the target list used
with "OVER window". As OVER only allows functions in PostgreSQL, I had
to make up a window function "rpr" which performs the row pattern
recognition task.  I was not able to find a way to implement
expressions like START.price (START is not a table alias). Any
suggestion is greatly appreciated.

The differences from the standard include:

o MEASURES is not supported
o SUBSET is not supported
o FIRST, LAST, CLASSIFIER are not supported
o PREV/NEXT in the standard accept more complex arguments
o Regular expressions other than "+" are not supported
o Only AFTER MATCH SKIP TO NEXT ROW is supported (if AFTER MATCH is
  not specified, AFTER MATCH SKIP TO NEXT ROW is assumed. In the
  standard AFTER MATCH SKIP PAST LAST ROW is assumed in this case). I
  have a plan to implement AFTER MATCH SKIP PAST LAST ROW though.
o INITIAL or SEEK are not supported ((behaves as if INITIAL is specified)
o Aggregate functions associated with window clause using RPR do not respect RPR

It seems RPR in the standard is quite complex. I think we can start
with a small subset of RPR then we could gradually enhance the
implementation.

Comments and suggestions are welcome.

[1] https://sqlperformance.com/2019/04/t-sql-queries/row-pattern-recognition-in-sql
[2] https://link.springer.com/article/10.1007/s13222-022-00404-3
[3] https://trino.io/docs/current/sql/pattern-recognition-in-window.html
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  2. Re: Row pattern recognition

    Vik Fearing <vik@postgresfriends.org> — 2023-06-25T21:08:35Z

    On 6/25/23 14:05, Tatsuo Ishii wrote:
> Attached is a PoC patch to implement "Row pattern recognition" (RPR)
> in SQL:2016 (I know SQL:2023 is already out, but I don't have access
> to it). Actually SQL:2016 defines RPR in two places[1]:
> 
>      Feature R010, “Row pattern recognition: FROM clause”
>      Feature R020, “Row pattern recognition: WINDOW clause”
> 
> The patch includes partial support for R020 part.

I have been dreaming of and lobbying for someone to pick up this 
feature.  I will be sure to review it from a standards perspective and 
will try my best to help with the technical aspect, but I am not sure to 
have the qualifications for that.

THANK YOU!

 > (I know SQL:2023 is already out, but I don't have access to it)

If you can, try to get ISO/IEC 19075-5 which is a guide to RPR instead 
of just its technical specification.

https://www.iso.org/standard/78936.html

> - What is RPR?
> 
> RPR provides a way to search series of data using regular expression
> patterns. Suppose you have a stock database.
> 
> CREATE TABLE stock (
>         company TEXT,
>         tdate DATE,
>         price BIGINT);
> 
> You want to find a "V-shaped" pattern: i.e. price goes down for 1 or
> more days, then goes up for 1 or more days. If we try to implement the
> query in PostgreSQL, it could be quite complex and inefficient.
> 
> RPR provides convenient way to implement the query.
> 
> SELECT company, tdate, price, rpr(price) OVER w FROM stock
>   WINDOW w AS (
>   PARTITION BY company
>   ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>   PATTERN (START DOWN+ UP+)
>   DEFINE
>    START AS TRUE,
>    UP AS price > PREV(price),
>    DOWN AS price < PREV(price)
> );
> 
> "PATTERN" and "DEFINE" are the key clauses of RPR. DEFINE defines 3
> "row pattern variables" namely START, UP and DOWN. They are associated
> with logical expressions namely "TRUE", "price > PREV(price)", and
> "price < PREV(price)". Note that "PREV" function returns price column
> in the previous row. So, UP is true if price is higher than previous
> day. On the other hand, DOWN is true if price is lower than previous
> day.  PATTERN uses the row pattern variables to create a necessary
> pattern.  In this case, the first row is always match because START is
> always true, and second or more rows match with "UP" ('+' is a regular
> expression representing one or more), and subsequent rows match with
> "DOWN".
> 
> Here is the sample output.
> 
>   company  |   tdate    | price | rpr
> ----------+------------+-------+------
>   company1 | 2023-07-01 |   100 |
>   company1 | 2023-07-02 |   200 |  200 -- 200->150->140->150
>   company1 | 2023-07-03 |   150 |  150 -- 150->140->150
>   company1 | 2023-07-04 |   140 |
>   company1 | 2023-07-05 |   150 |  150 -- 150->90->110->130
>   company1 | 2023-07-06 |    90 |
>   company1 | 2023-07-07 |   110 |
>   company1 | 2023-07-08 |   130 |
>   company1 | 2023-07-09 |   120 |
>   company1 | 2023-07-10 |   130 |
> 
> rpr shows the first row if all the patterns are satisfied. In the
> example above 200, 150, 150 are the cases.  Other rows are shown as
> NULL. For example, on 2023-07-02 price starts with 200, then goes down
> to 150 then 140 but goes up 150 next day.

I don't understand this.  RPR in a window specification limits the 
window to the matched rows, so this looks like your rpr() function is 
just the regular first_value() window function that we already have?

> As far as I know, only Oracle implements RPR (only R010. R020 is not
> implemented) among OSS/commercial RDBMSs. There are a few DWH software
> having RPR. According to [2] they are Snowflake and MS Stream
> Analytics. It seems Trino is another one[3].

I thought DuckDB had it already, but it looks like I was wrong.

> - Note about the patch
> 
> The current implementation is not only a subset of the standard, but
> is different from it in some places. The example above is written as
> follows according to the standard:
> 
> SELECT company, tdate, startprice OVER w FROM stock
>   WINDOW w AS (
>   PARTITION BY company
>   MEASURES
>    START.price AS startprice
>   ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>   PATTERN (START DOWN+ UP+)
>   DEFINE
>    START AS TRUE,
>    UP AS UP.price > PREV(UP.price),
>    DOWN AS DOWN.price < PREV(DOWN.price)
> );
> 
> Notice that rpr(price) is written as START.price and startprice in the
> standard. MEASURES defines variable names used in the target list used
> with "OVER window". As OVER only allows functions in PostgreSQL, I had
> to make up a window function "rpr" which performs the row pattern
> recognition task.  I was not able to find a way to implement
> expressions like START.price (START is not a table alias). Any
> suggestion is greatly appreciated.

As in your example, you cannot have START.price outside of the window 
specification; it can only go in the MEASURES clause.  Only startprice 
is allowed outside and it gets its qualification from the OVER.  Using 
w.startprice might have been better but that would require window names 
to be in the same namespace as range tables.

This currently works in Postgres:

   SELECT RANK() OVER w
   FROM (VALUES (1)) AS w (x)
   WINDOW w AS (ORDER BY w.x);

> The differences from the standard include:
> 
> o MEASURES is not supported
 > o FIRST, LAST, CLASSIFIER are not supported
 > o PREV/NEXT in the standard accept more complex arguments
 > o INITIAL or SEEK are not supported ((behaves as if INITIAL is specified)

Okay, for now.

> o SUBSET is not supported

Is this because you haven't done it yet, or because you ran into 
problems trying to do it?

> o Regular expressions other than "+" are not supported

This is what I had a hard time imagining how to do while thinking about 
it.  The grammar is so different here and we allow many more operators 
(like "?" which is also the standard parameter symbol).  People more 
expert than me will have to help here.

> o Only AFTER MATCH SKIP TO NEXT ROW is supported (if AFTER MATCH is
>    not specified, AFTER MATCH SKIP TO NEXT ROW is assumed. In the
>    standard AFTER MATCH SKIP PAST LAST ROW is assumed in this case). I
>    have a plan to implement AFTER MATCH SKIP PAST LAST ROW though.

In this case, we should require the user to specify AFTER MATCH SKIP TO 
NEXT ROW so that behavior doesn't change when we implement the standard 
default.  (Your patch might do this already.)

> o Aggregate functions associated with window clause using RPR do not respect RPR

I do not understand what this means.

> It seems RPR in the standard is quite complex. I think we can start
> with a small subset of RPR then we could gradually enhance the
> implementation.

I have no problem with that as long as we don't paint ourselves into a 
corner.

> Comments and suggestions are welcome.

I have not looked at the patch yet, but is the reason for doing R020 
before R010 because you haven't done the MEASURES clause yet?

In any case, I will be watching this with a close eye, and I am eager to 
help in any way I can.
-- 
Vik Fearing

  3. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-06-26T01:05:20Z

    > I have been dreaming of and lobbying for someone to pick up this
> feature.  I will be sure to review it from a standards perspective and
> will try my best to help with the technical aspect, but I am not sure
> to have the qualifications for that.
> 
> THANK YOU!

Thank you for looking into my proposal.

>> (I know SQL:2023 is already out, but I don't have access to it)
> 
> If you can, try to get ISO/IEC 19075-5 which is a guide to RPR instead
> of just its technical specification.
> 
> https://www.iso.org/standard/78936.html

Thanks for the info.

> I don't understand this.  RPR in a window specification limits the
> window to the matched rows, so this looks like your rpr() function is
> just the regular first_value() window function that we already have?

No, rpr() is different from first_value(). rpr() returns the argument
value at the first row in a frame only when matched rows found. On the
other hand first_value() returns the argument value at the first row
in a frame unconditionally.

company  |   tdate    | price | rpr  | first_value 
----------+------------+-------+------+-------------
 company1 | 2023-07-01 |   100 |      |         100
 company1 | 2023-07-02 |   200 |  200 |         200
 company1 | 2023-07-03 |   150 |  150 |         150
 company1 | 2023-07-04 |   140 |      |         140
 company1 | 2023-07-05 |   150 |  150 |         150
 company1 | 2023-07-06 |    90 |      |          90
 company1 | 2023-07-07 |   110 |      |         110
 company1 | 2023-07-08 |   130 |      |         130
 company1 | 2023-07-09 |   120 |      |         120
 company1 | 2023-07-10 |   130 |      |         130

For example, a frame starting with (tdate = 2023-07-02, price = 200)
consists of rows (price = 200, 150, 140, 150) satisfying the pattern,
thus rpr returns 200. Since in this example frame option "ROWS BETWEEN
CURRENT ROW AND UNBOUNDED FOLLOWING" is specified, next frame starts
with (tdate = 2023-07-03, price = 150). This frame satisfies the
pattern too (price = 150, 140, 150), and rpr retus 150... and so on.

> As in your example, you cannot have START.price outside of the window
> specification; it can only go in the MEASURES clause.  Only startprice
> is allowed outside and it gets its qualification from the OVER.  Using
> w.startprice might have been better but that would require window
> names to be in the same namespace as range tables.
> 
> This currently works in Postgres:
> 
>   SELECT RANK() OVER w
>   FROM (VALUES (1)) AS w (x)
>   WINDOW w AS (ORDER BY w.x);

Interesting.

>> o SUBSET is not supported
> 
> Is this because you haven't done it yet, or because you ran into
> problems trying to do it?

Because it seems SUBSET is not useful without MEASURES support. Thus
my plan is, firstly implement MEASURES, then SUBSET. What do you
think?

>> o Regular expressions other than "+" are not supported
> 
> This is what I had a hard time imagining how to do while thinking
> about it.  The grammar is so different here and we allow many more
> operators (like "?" which is also the standard parameter symbol).
> People more expert than me will have to help here.

Yes, that is a problem.

> In this case, we should require the user to specify AFTER MATCH SKIP
> TO NEXT ROW so that behavior doesn't change when we implement the
> standard default.  (Your patch might do this already.)

Agreed. I will implement AFTER MATCH SKIP PAST LAST ROW in the next
patch and I will change the default to AFTER MATCH SKIP PAST LAST ROW.

>> o Aggregate functions associated with window clause using RPR do not
>> respect RPR
> 
> I do not understand what this means.

Ok, let me explain. See example below. In my understanding "count"
should retun the number of rows in a frame restriced by the match
condition. For example at the first line (2023-07-01 | 100) count
returns 10. I think this should be 0 because the "restriced" frame
starting at the line contains no matched row. On the other hand the
(restricted) frame starting at second line (2023-07-02 | 200) contains
4 rows, thus count should return 4, instead of 9.

SELECT company, tdate, price, rpr(price) OVER w, count(*) OVER w FROM stock
 WINDOW w AS (
 PARTITION BY company
 ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
 PATTERN (START DOWN+ UP+)
 DEFINE
  START AS TRUE,
  UP AS price > PREV(price),
  DOWN AS price < PREV(price)
);

company  |   tdate    | price | rpr  | count 
----------+------------+-------+------+-------
 company1 | 2023-07-01 |   100 |      |    10
 company1 | 2023-07-02 |   200 |  200 |     9
 company1 | 2023-07-03 |   150 |  150 |     8
 company1 | 2023-07-04 |   140 |      |     7
 company1 | 2023-07-05 |   150 |  150 |     6
 company1 | 2023-07-06 |    90 |      |     5
 company1 | 2023-07-07 |   110 |      |     4
 company1 | 2023-07-08 |   130 |      |     3
 company1 | 2023-07-09 |   120 |      |     2
 company1 | 2023-07-10 |   130 |      |     1

>> It seems RPR in the standard is quite complex. I think we can start
>> with a small subset of RPR then we could gradually enhance the
>> implementation.
> 
> I have no problem with that as long as we don't paint ourselves into a
> corner.

Totally agreed.

>> Comments and suggestions are welcome.
> 
> I have not looked at the patch yet, but is the reason for doing R020
> before R010 because you haven't done the MEASURES clause yet?

One of the reasons is, implementing MATCH_RECOGNIZE (R010) looked
harder for me because modifying main SELECT clause could be a hard
work. Another reason is, I had no idea how to implement PREV/NEXT in
other than in WINDOW clause. Other people might feel differently
though.

> In any case, I will be watching this with a close eye, and I am eager
> to help in any way I can.

Thank you! I am looking forward to comments on my patch.  Also any
idea how to implement MEASURES clause is welcome.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  4. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-06-26T08:45:07Z

    >> In this case, we should require the user to specify AFTER MATCH SKIP
>> TO NEXT ROW so that behavior doesn't change when we implement the
>> standard default.  (Your patch might do this already.)
> 
> Agreed. I will implement AFTER MATCH SKIP PAST LAST ROW in the next
> patch and I will change the default to AFTER MATCH SKIP PAST LAST ROW.

Attached is the v2 patch to add support for AFTER MATCH SKIP PAST LAST
ROW and AFTER MATCH SKIP PAST LAST ROW. The default is AFTER MATCH
SKIP PAST LAST ROW as the standard default. Here are some examples to
demonstrate how those clauses affect the query result.

SELECT i, rpr(i) OVER w
  FROM (VALUES (1), (2), (3), (4)) AS v (i)
  WINDOW w AS (
   ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
   AFTER MATCH SKIP PAST LAST ROW
   PATTERN (A B)
   DEFINE
    A AS i <= 2,
    B AS i <= 3
);
 i | rpr 
---+-----
 1 |   1
 2 |    
 3 |    
 4 |    
(4 rows)

In this example rpr starts from i = 1 and find that row i = 1
satisfies A, and row i = 2 satisfies B. Then rpr moves to row i = 3
and find that it does not satisfy A, thus the result is NULL. Same
thing can be said to row i = 4.

SELECT i, rpr(i) OVER w
  FROM (VALUES (1), (2), (3), (4)) AS v (i)
  WINDOW w AS (
   ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
   AFTER MATCH SKIP TO NEXT ROW
   PATTERN (A B)
   DEFINE
    A AS i <= 2,
    B AS i <= 3
);
 i | rpr 
---+-----
 1 |   1
 2 |   2
 3 |    
 4 |    
(4 rows)

In this example rpr starts from i = 1 and find that row i = 1
satisfies A, and row i = 2 satisfies B (same as above). Then rpr moves
to row i = 2, rather than 3 because AFTER MATCH SKIP TO NEXT ROW is
specified.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  5. Re: Row pattern recognition

    Vik Fearing <vik@postgresfriends.org> — 2023-06-26T22:38:20Z

    On 6/26/23 03:05, Tatsuo Ishii wrote:
>> I don't understand this.  RPR in a window specification limits the
>> window to the matched rows, so this looks like your rpr() function is
>> just the regular first_value() window function that we already have?
> 
> No, rpr() is different from first_value(). rpr() returns the argument
> value at the first row in a frame only when matched rows found. On the
> other hand first_value() returns the argument value at the first row
> in a frame unconditionally.
> 
> company  |   tdate    | price | rpr  | first_value
> ----------+------------+-------+------+-------------
>   company1 | 2023-07-01 |   100 |      |         100
>   company1 | 2023-07-02 |   200 |  200 |         200
>   company1 | 2023-07-03 |   150 |  150 |         150
>   company1 | 2023-07-04 |   140 |      |         140
>   company1 | 2023-07-05 |   150 |  150 |         150
>   company1 | 2023-07-06 |    90 |      |          90
>   company1 | 2023-07-07 |   110 |      |         110
>   company1 | 2023-07-08 |   130 |      |         130
>   company1 | 2023-07-09 |   120 |      |         120
>   company1 | 2023-07-10 |   130 |      |         130
> 
> For example, a frame starting with (tdate = 2023-07-02, price = 200)
> consists of rows (price = 200, 150, 140, 150) satisfying the pattern,
> thus rpr returns 200. Since in this example frame option "ROWS BETWEEN
> CURRENT ROW AND UNBOUNDED FOLLOWING" is specified, next frame starts
> with (tdate = 2023-07-03, price = 150). This frame satisfies the
> pattern too (price = 150, 140, 150), and rpr retus 150... and so on.


Okay, I see the problem now, and why you need the rpr() function.

You are doing this as something that happens over a window frame, but it 
is actually something that *reduces* the window frame.  The pattern 
matching needs to be done when the frame is calculated and not when any 
particular function is applied over it.

This query (with all the defaults made explicit):

SELECT s.company, s.tdate, s.price,
        FIRST_VALUE(s.tdate) OVER w,
        LAST_VALUE(s.tdate) OVER w,
        lowest OVER w
FROM stock AS s
WINDOW w AS (
   PARTITION BY s.company
   ORDER BY s.tdate
   ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
   EXCLUDE NO OTHERS
   MEASURES
     LAST(DOWN) AS lowest
   AFTER MATCH SKIP PAST LAST ROW
   INITIAL PATTERN (START DOWN+ UP+)
   DEFINE
     START AS TRUE,
     UP AS price > PREV(price),
     DOWN AS price < PREV(price)
);

Should produce this result:

  company  |   tdate    | price | first_value | last_value | lowest
----------+------------+-------+-------------+------------+--------
  company1 | 07-01-2023 |   100 |             |            |
  company1 | 07-02-2023 |   200 | 07-02-2023  | 07-05-2023 |    140
  company1 | 07-03-2023 |   150 |             |            |
  company1 | 07-04-2023 |   140 |             |            |
  company1 | 07-05-2023 |   150 |             |            |
  company1 | 07-06-2023 |    90 |             |            |
  company1 | 07-07-2023 |   110 |             |            |
  company1 | 07-08-2023 |   130 | 07-05-2023  | 07-05-2023 |    120
  company1 | 07-09-2023 |   120 |             |            |
  company1 | 07-10-2023 |   130 |             |            |
(10 rows)

Or if we switch to AFTER MATCH SKIP TO NEXT ROW, then we get:

  company  |   tdate    | price | first_value | last_value | lowest
----------+------------+-------+-------------+------------+--------
  company1 | 07-01-2023 |   100 |             |            |
  company1 | 07-02-2023 |   200 | 07-02-2023  | 07-05-2023 |    140
  company1 | 07-03-2023 |   150 | 07-03-2023  | 07-05-2023 |    140
  company1 | 07-04-2023 |   140 |             |            |
  company1 | 07-05-2023 |   150 | 07-05-2023  | 07-08-2023 |     90
  company1 | 07-06-2023 |    90 |             |            |
  company1 | 07-07-2023 |   110 |             |            |
  company1 | 07-08-2023 |   130 | 07-08-2023  | 07-10-2023 |    120
  company1 | 07-09-2023 |   120 |             |            |
  company1 | 07-10-2023 |   130 |             |            |
(10 rows)

And then if we change INITIAL to SEEK:

  company  |   tdate    | price | first_value | last_value | lowest
----------+------------+-------+-------------+------------+--------
  company1 | 07-01-2023 |   100 | 07-02-2023  | 07-05-2023 |    140
  company1 | 07-02-2023 |   200 | 07-02-2023  | 07-05-2023 |    140
  company1 | 07-03-2023 |   150 | 07-03-2023  | 07-05-2023 |    140
  company1 | 07-04-2023 |   140 | 07-05-2023  | 07-08-2023 |     90
  company1 | 07-05-2023 |   150 | 07-05-2023  | 07-08-2023 |     90
  company1 | 07-06-2023 |    90 | 07-08-2023  | 07-10-2023 |    120
  company1 | 07-07-2023 |   110 | 07-08-2023  | 07-10-2023 |    120
  company1 | 07-08-2023 |   130 | 07-08-2023  | 07-10-2023 |    120
  company1 | 07-09-2023 |   120 |             |            |
  company1 | 07-10-2023 |   130 |             |            |
(10 rows)

Since the pattern recognition is part of the frame, the window 
aggregates should Just Work.


>>> o SUBSET is not supported
>>
>> Is this because you haven't done it yet, or because you ran into
>> problems trying to do it?
> 
> Because it seems SUBSET is not useful without MEASURES support. Thus
> my plan is, firstly implement MEASURES, then SUBSET. What do you
> think?


SUBSET elements can be used in DEFINE clauses, but I do not think this 
is important compared to other features.


>>> Comments and suggestions are welcome.
>>
>> I have not looked at the patch yet, but is the reason for doing R020
>> before R010 because you haven't done the MEASURES clause yet?
> 
> One of the reasons is, implementing MATCH_RECOGNIZE (R010) looked
> harder for me because modifying main SELECT clause could be a hard
> work. Another reason is, I had no idea how to implement PREV/NEXT in
> other than in WINDOW clause. Other people might feel differently
> though.


I think we could do this with a single tuplesort if we use backtracking 
(which might be really slow for some patterns).  I have not looked into 
it in any detail.

We would need to be able to remove tuples from the end (even if only 
logically), and be able to update tuples inside the store.  Both of 
those needs come from backtracking and possibly changing the classifier.

Without backtracking, I don't see how we could do it without have a 
separate tuplestore for every current possible match.


>> In any case, I will be watching this with a close eye, and I am eager
>> to help in any way I can.
> 
> Thank you! I am looking forward to comments on my patch.  Also any
> idea how to implement MEASURES clause is welcome.


I looked at your v2 patches a little bit and the only comment that I 
currently have on the code is you spelled PERMUTE as PREMUTE. 
Everything else is hopefully explained above.
-- 
Vik Fearing

  6. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-06-28T00:58:19Z

    > Okay, I see the problem now, and why you need the rpr() function.
> 
> You are doing this as something that happens over a window frame, but
> it is actually something that *reduces* the window frame.  The pattern
> matching needs to be done when the frame is calculated and not when
> any particular function is applied over it.

Yes. (I think the standard calls the window frame as "full window
frame" in context of RPR to make a contrast with the subset of the
frame rows restricted by RPR. The paper I refered to as [2] claims
that the latter window frame is called "reduced window frame" in the
standard but I wasn't able to find the term in the standard.)

I wanted to demonstate that pattern matching logic is basically
correct in the PoC patch. Now what I need to do is, move the row
pattern matching logic to somewhere inside nodeWindowAgg so that
"restricted window frame" can be applied to all window functions and
window aggregates. Currently I am looking into update_frameheadpos()
and update_frametailpos() which calculate the frame head and tail
against current row. What do you think?

> This query (with all the defaults made explicit):
> 
> SELECT s.company, s.tdate, s.price,
>        FIRST_VALUE(s.tdate) OVER w,
>        LAST_VALUE(s.tdate) OVER w,
>        lowest OVER w
> FROM stock AS s
> WINDOW w AS (
>   PARTITION BY s.company
>   ORDER BY s.tdate
>   ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>   EXCLUDE NO OTHERS
>   MEASURES
>     LAST(DOWN) AS lowest
>   AFTER MATCH SKIP PAST LAST ROW
>   INITIAL PATTERN (START DOWN+ UP+)
>   DEFINE
>     START AS TRUE,
>     UP AS price > PREV(price),
>     DOWN AS price < PREV(price)
> );
> 
> Should produce this result:

[snip]

Thanks for the examples. I agree with the expected query results.

>>>> o SUBSET is not supported
>>>
>>> Is this because you haven't done it yet, or because you ran into
>>> problems trying to do it?
>> Because it seems SUBSET is not useful without MEASURES support. Thus
>> my plan is, firstly implement MEASURES, then SUBSET. What do you
>> think?
> 
> 
> SUBSET elements can be used in DEFINE clauses, but I do not think this
> is important compared to other features.

Ok.

>>> I have not looked at the patch yet, but is the reason for doing R020
>>> before R010 because you haven't done the MEASURES clause yet?
>> One of the reasons is, implementing MATCH_RECOGNIZE (R010) looked
>> harder for me because modifying main SELECT clause could be a hard
>> work. Another reason is, I had no idea how to implement PREV/NEXT in
>> other than in WINDOW clause. Other people might feel differently
>> though.
> 
> 
> I think we could do this with a single tuplesort if we use
> backtracking (which might be really slow for some patterns).  I have
> not looked into it in any detail.
> 
> We would need to be able to remove tuples from the end (even if only
> logically), and be able to update tuples inside the store.  Both of
> those needs come from backtracking and possibly changing the
> classifier.
> 
> Without backtracking, I don't see how we could do it without have a
> separate tuplestore for every current possible match.

Maybe an insane idea but what about rewriting MATCH_RECOGNIZE clause
into Window clause with RPR?

> I looked at your v2 patches a little bit and the only comment that I
> currently have on the code is you spelled PERMUTE as
> PREMUTE. Everything else is hopefully explained above.

Thanks. Will fix.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  7. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-06-28T12:17:00Z

    Small question.

> This query (with all the defaults made explicit):
> 
> SELECT s.company, s.tdate, s.price,
>        FIRST_VALUE(s.tdate) OVER w,
>        LAST_VALUE(s.tdate) OVER w,
>        lowest OVER w
> FROM stock AS s
> WINDOW w AS (
>   PARTITION BY s.company
>   ORDER BY s.tdate
>   ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>   EXCLUDE NO OTHERS
>   MEASURES
>     LAST(DOWN) AS lowest
>   AFTER MATCH SKIP PAST LAST ROW
>   INITIAL PATTERN (START DOWN+ UP+)
>   DEFINE
>     START AS TRUE,
>     UP AS price > PREV(price),
>     DOWN AS price < PREV(price)
> );

>     LAST(DOWN) AS lowest

should be "LAST(DOWN.price) AS lowest"?

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  8. Re: Row pattern recognition

    Vik Fearing <vik@postgresfriends.org> — 2023-06-28T22:30:43Z

    On 6/28/23 14:17, Tatsuo Ishii wrote:
> Small question.
> 
>> This query (with all the defaults made explicit):
>>
>> SELECT s.company, s.tdate, s.price,
>>         FIRST_VALUE(s.tdate) OVER w,
>>         LAST_VALUE(s.tdate) OVER w,
>>         lowest OVER w
>> FROM stock AS s
>> WINDOW w AS (
>>    PARTITION BY s.company
>>    ORDER BY s.tdate
>>    ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>>    EXCLUDE NO OTHERS
>>    MEASURES
>>      LAST(DOWN) AS lowest
>>    AFTER MATCH SKIP PAST LAST ROW
>>    INITIAL PATTERN (START DOWN+ UP+)
>>    DEFINE
>>      START AS TRUE,
>>      UP AS price > PREV(price),
>>      DOWN AS price < PREV(price)
>> );
> 
>>      LAST(DOWN) AS lowest
> 
> should be "LAST(DOWN.price) AS lowest"?

Yes, it should be.  And the tdate='07-08-2023' row in the first 
resultset should have '07-08-2023' and '07-10-2023' as its 4th and 5th 
columns.

Since my brain is doing the processing instead of postgres, I made some 
human errors. :-)
-- 
Vik Fearing

  9. Re: Row pattern recognition

    Jacob Champion <jchampion@timescale.com> — 2023-07-19T16:30:40Z

    Hello,

Thanks for working on this! We're interested in RPR as well, and I've
been trying to get up to speed with the specs, to maybe make myself
useful.

On 6/27/23 17:58, Tatsuo Ishii wrote:
> Yes. (I think the standard calls the window frame as "full window
> frame" in context of RPR to make a contrast with the subset of the
> frame rows restricted by RPR. The paper I refered to as [2] claims
> that the latter window frame is called "reduced window frame" in the
> standard but I wasn't able to find the term in the standard.)

19075-5 discusses that, at least; not sure about other parts of the spec.

> Maybe an insane idea but what about rewriting MATCH_RECOGNIZE clause
> into Window clause with RPR?

Are we guaranteed to always have an equivalent window clause? There seem
to be many differences between the two, especially when it comes to ONE
ROW/ALL ROWS PER MATCH.

--

To add onto what Vik said above:

>> It seems RPR in the standard is quite complex. I think we can start
>> with a small subset of RPR then we could gradually enhance the
>> implementation.
> 
> I have no problem with that as long as we don't paint ourselves into a 
> corner.

To me, PATTERN looks like an area where we may want to support a broader
set of operations in the first version. The spec has a bunch of
discussion around cases like empty matches, match order of alternation
and permutation, etc., which are not possible to express or test with
only the + quantifier. Those might be harder to get right in a v2, if we
don't at least keep them in mind for v1?

> +static List *
> +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef)
> +{
> +   List        *patterns;

My compiler complains about the `patterns` variable here, which is
returned without ever being initialized. (The caller doesn't seem to use
it.)

> +-- basic test using PREV
> +SELECT company, tdate, price, rpr(price) OVER w FROM stock
> + WINDOW w AS (
> + PARTITION BY company
> + ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
> + INITIAL
> + PATTERN (START UP+ DOWN+)
> + DEFINE
> +  START AS TRUE,
> +  UP AS price > PREV(price),
> +  DOWN AS price < PREV(price)
> +);

nitpick: IMO the tests should be making use of ORDER BY in the window
clauses.

This is a very big feature. I agree with you that MEASURES seems like a
very important "next piece" to add. Are there other areas where you
would like reviewers to focus on right now (or avoid)?

Thanks!
--Jacob

  10. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-07-20T05:15:13Z

    > Hello,
> 
> Thanks for working on this! We're interested in RPR as well, and I've
> been trying to get up to speed with the specs, to maybe make myself
> useful.

Thank you for being interested in this.

> 19075-5 discusses that, at least; not sure about other parts of the spec.

Thanks for the info. Unfortunately I don't have 19075-5 though.

>> Maybe an insane idea but what about rewriting MATCH_RECOGNIZE clause
>> into Window clause with RPR?
> 
> Are we guaranteed to always have an equivalent window clause? There seem
> to be many differences between the two, especially when it comes to ONE
> ROW/ALL ROWS PER MATCH.

You are right. I am not 100% sure if the rewriting is possible at this
point.

> To add onto what Vik said above:
> 
>>> It seems RPR in the standard is quite complex. I think we can start
>>> with a small subset of RPR then we could gradually enhance the
>>> implementation.
>> 
>> I have no problem with that as long as we don't paint ourselves into a 
>> corner.
> 
> To me, PATTERN looks like an area where we may want to support a broader
> set of operations in the first version.

Me too but...

> The spec has a bunch of
> discussion around cases like empty matches, match order of alternation
> and permutation, etc., which are not possible to express or test with
> only the + quantifier. Those might be harder to get right in a v2, if we
> don't at least keep them in mind for v1?

Currently my patch has a limitation for the sake of simple
implementation: a pattern like "A+" is parsed and analyzed in the raw
parser. This makes subsequent process much easier because the pattern
element variable (in this case "A") and the quantifier (in this case
"+") is already identified by the raw parser. However there are much
more cases are allowed in the standard as you already pointed out. For
those cases probably we should give up to parse PATTERN items in the
raw parser, and instead the raw parser just accepts the elements as
Sconst?

>> +static List *
>> +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef)
>> +{
>> +   List        *patterns;
> 
> My compiler complains about the `patterns` variable here, which is
> returned without ever being initialized. (The caller doesn't seem to use
> it.)

Will fix.

>> +-- basic test using PREV
>> +SELECT company, tdate, price, rpr(price) OVER w FROM stock
>> + WINDOW w AS (
>> + PARTITION BY company
>> + ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>> + INITIAL
>> + PATTERN (START UP+ DOWN+)
>> + DEFINE
>> +  START AS TRUE,
>> +  UP AS price > PREV(price),
>> +  DOWN AS price < PREV(price)
>> +);
> 
> nitpick: IMO the tests should be making use of ORDER BY in the window
> clauses.

Right. Will fix.

> This is a very big feature. I agree with you that MEASURES seems like a
> very important "next piece" to add. Are there other areas where you
> would like reviewers to focus on right now (or avoid)?

Any comments, especially on the PREV/NEXT implementation part is
welcome. Currently the DEFINE expression like "price > PREV(price)" is
prepared in ExecInitWindowAgg using ExecInitExpr,tweaking var->varno
in Var node so that PREV uses OUTER_VAR, NEXT uses INNER_VAR.  Then
evaluate the expression in ExecWindowAgg using ExecEvalExpr, setting
previous row TupleSlot to ExprContext->ecxt_outertuple, and next row
TupleSlot to ExprContext->ecxt_innertuple. I think this is temporary
hack and should be gotten ride of before v1 is committed. Better idea?

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  11. Re: Row pattern recognition

    Jacob Champion <jchampion@timescale.com> — 2023-07-20T23:36:37Z

    Hi Ishii-san,

On 7/19/23 22:15, Tatsuo Ishii wrote:
> Currently my patch has a limitation for the sake of simple
> implementation: a pattern like "A+" is parsed and analyzed in the raw
> parser. This makes subsequent process much easier because the pattern
> element variable (in this case "A") and the quantifier (in this case
> "+") is already identified by the raw parser. However there are much
> more cases are allowed in the standard as you already pointed out. For
> those cases probably we should give up to parse PATTERN items in the
> raw parser, and instead the raw parser just accepts the elements as
> Sconst?

Is there a concern that the PATTERN grammar can't be represented in
Bison? I thought it was all context-free... Or is the concern that the
parse tree of the pattern is hard to feed into a regex engine?

> Any comments, especially on the PREV/NEXT implementation part is
> welcome. Currently the DEFINE expression like "price > PREV(price)" is
> prepared in ExecInitWindowAgg using ExecInitExpr,tweaking var->varno
> in Var node so that PREV uses OUTER_VAR, NEXT uses INNER_VAR.  Then
> evaluate the expression in ExecWindowAgg using ExecEvalExpr, setting
> previous row TupleSlot to ExprContext->ecxt_outertuple, and next row
> TupleSlot to ExprContext->ecxt_innertuple. I think this is temporary
> hack and should be gotten ride of before v1 is committed. Better idea?

I'm not familiar enough with this code yet to offer very concrete
suggestions, sorry... But at some point in the future, we need to be
able to skip forward and backward from arbitrary points, like

    DEFINE B AS B.price > PREV(FIRST(A.price), 3)

so there won't be just one pair of "previous and next" tuples. Maybe
that can help clarify the design? It feels like it'll need to eventually
be a "real" function that operates on the window state, even if it
doesn't support all of the possible complexities in v1.

--

Taking a closer look at the regex engine:

It looks like the + qualifier has trouble when it meets the end of the
frame. For instance, try removing the last row of the 'stock' table in
rpr.sql; some of the final matches will disappear unexpectedly. Or try a
pattern like

    PATTERN ( a+ )
     DEFINE a AS TRUE

which doesn't seem to match anything in my testing.

There's also the issue of backtracking in the face of reclassification,
as I think Vik was alluding to upthread. The pattern

    PATTERN ( a+ b+ )
     DEFINE a AS col = 2,
            b AS col = 2

doesn't match a sequence of values (2 2 ...) with the current
implementation, even with a dummy row at the end to avoid the
end-of-frame bug.

(I've attached two failing tests against v2, to hopefully better
illustrate, along with what I _think_ should be the correct results.)

I'm not quite understanding the match loop in evaluate_pattern(). It
looks like we're building up a string to pass to the regex engine, but
by the we call regexp_instr, don't we already know whether or not the
pattern will match based on the expression evaluation we've done?

Thanks,
--Jacob

  12. Re: Row pattern recognition

    Vik Fearing <vik@postgresfriends.org> — 2023-07-21T00:07:44Z

    On 7/21/23 01:36, Jacob Champion wrote:
> There's also the issue of backtracking in the face of reclassification,
> as I think Vik was alluding to upthread.

We definitely need some kind of backtracking or other reclassification 
method.

> (I've attached two failing tests against v2, to hopefully better
> illustrate, along with what I_think_  should be the correct results.)

Almost.  You are matching 07-01-2023 but the condition is "price > 100".
-- 
Vik Fearing

  13. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-07-21T06:16:48Z

    Hi,

> Hi Ishii-san,
> 
> On 7/19/23 22:15, Tatsuo Ishii wrote:
>> Currently my patch has a limitation for the sake of simple
>> implementation: a pattern like "A+" is parsed and analyzed in the raw
>> parser. This makes subsequent process much easier because the pattern
>> element variable (in this case "A") and the quantifier (in this case
>> "+") is already identified by the raw parser. However there are much
>> more cases are allowed in the standard as you already pointed out. For
>> those cases probably we should give up to parse PATTERN items in the
>> raw parser, and instead the raw parser just accepts the elements as
>> Sconst?
> 
> Is there a concern that the PATTERN grammar can't be represented in
> Bison? I thought it was all context-free...

I don't know at this point. I think context-free is not enough to be
repsented in Bison. The grammer also needs to be LALR(1).  Moreover,
adding the grammer to existing parser may generate shift/reduce
errors.

> Or is the concern that the
> parse tree of the pattern is hard to feed into a regex engine?

One small concern is how to convert pattern variables to regex
expression which our regex enginer understands. Suppose,

PATTERN UP+

Currently I convert "UP+" to "U+" so that it can be fed to the regexp
engine. In order to do that, we need to know which part of the pattern
(UP+) is the pattern variable ("UP"). For "UP+" it's quite easy. But
for more complex regular expressions it would be not, unless PATTERN
grammer can be analyzed by our parser to know which part is the
pattern variable.

>> Any comments, especially on the PREV/NEXT implementation part is
>> welcome. Currently the DEFINE expression like "price > PREV(price)" is
>> prepared in ExecInitWindowAgg using ExecInitExpr,tweaking var->varno
>> in Var node so that PREV uses OUTER_VAR, NEXT uses INNER_VAR.  Then
>> evaluate the expression in ExecWindowAgg using ExecEvalExpr, setting
>> previous row TupleSlot to ExprContext->ecxt_outertuple, and next row
>> TupleSlot to ExprContext->ecxt_innertuple. I think this is temporary
>> hack and should be gotten ride of before v1 is committed. Better idea?
> 
> I'm not familiar enough with this code yet to offer very concrete
> suggestions, sorry... But at some point in the future, we need to be
> able to skip forward and backward from arbitrary points, like
> 
>     DEFINE B AS B.price > PREV(FIRST(A.price), 3)
> 
> so there won't be just one pair of "previous and next" tuples.

Yes, I know.

> Maybe
> that can help clarify the design? It feels like it'll need to eventually
> be a "real" function that operates on the window state, even if it
> doesn't support all of the possible complexities in v1.

Unfortunately an window function can not call other window functions.

> Taking a closer look at the regex engine:
> 
> It looks like the + qualifier has trouble when it meets the end of the
> frame. For instance, try removing the last row of the 'stock' table in
> rpr.sql; some of the final matches will disappear unexpectedly. Or try a
> pattern like
> 
>     PATTERN ( a+ )
>      DEFINE a AS TRUE
> 
> which doesn't seem to match anything in my testing.
> 
> There's also the issue of backtracking in the face of reclassification,
> as I think Vik was alluding to upthread. The pattern
> 
>     PATTERN ( a+ b+ )
>      DEFINE a AS col = 2,
>             b AS col = 2
> 
> doesn't match a sequence of values (2 2 ...) with the current
> implementation, even with a dummy row at the end to avoid the
> end-of-frame bug.
> 
> (I've attached two failing tests against v2, to hopefully better
> illustrate, along with what I _think_ should be the correct results.)

Thanks. I will look into this.

> I'm not quite understanding the match loop in evaluate_pattern(). It
> looks like we're building up a string to pass to the regex engine, but
> by the we call regexp_instr, don't we already know whether or not the
> pattern will match based on the expression evaluation we've done?

For "+" yes. But for more complex regular expression like '{n}', we
need to call our regexp engine to check if the pattern matches.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  14. Re: Row pattern recognition

    Jacob Champion <jchampion@timescale.com> — 2023-07-21T23:14:12Z

    On 7/20/23 17:07, Vik Fearing wrote:
> On 7/21/23 01:36, Jacob Champion wrote:
>> (I've attached two failing tests against v2, to hopefully better
>> illustrate, along with what I_think_  should be the correct results.)
> 
> Almost.  You are matching 07-01-2023 but the condition is "price > 100".

D'oh. Correction attached. I think :)

Thanks,
--Jacob

  15. Re: Row pattern recognition

    Jacob Champion <jchampion@timescale.com> — 2023-07-21T23:16:18Z

    On 7/20/23 23:16, Tatsuo Ishii wrote:
> I don't know at this point. I think context-free is not enough to be
> repsented in Bison. The grammer also needs to be LALR(1).  Moreover,
> adding the grammer to existing parser may generate shift/reduce
> errors.

Ah. It's been too long since my compilers classes; I will pipe down.

> One small concern is how to convert pattern variables to regex
> expression which our regex enginer understands. Suppose,
> 
> PATTERN UP+
> 
> Currently I convert "UP+" to "U+" so that it can be fed to the regexp
> engine. In order to do that, we need to know which part of the pattern
> (UP+) is the pattern variable ("UP"). For "UP+" it's quite easy. But
> for more complex regular expressions it would be not, unless PATTERN
> grammer can be analyzed by our parser to know which part is the
> pattern variable.

Is the eventual plan to generate multiple alternatives, and run the
regex against them one at a time?

>> I'm not familiar enough with this code yet to offer very concrete
>> suggestions, sorry... But at some point in the future, we need to be
>> able to skip forward and backward from arbitrary points, like
>>
>>     DEFINE B AS B.price > PREV(FIRST(A.price), 3)
>>
>> so there won't be just one pair of "previous and next" tuples.
> 
> Yes, I know.

I apologize. I got overexplain-y.

>> Maybe
>> that can help clarify the design? It feels like it'll need to eventually
>> be a "real" function that operates on the window state, even if it
>> doesn't support all of the possible complexities in v1.
> 
> Unfortunately an window function can not call other window functions.

Can that restriction be lifted for the EXPR_KIND_RPR_DEFINE case? Or
does it make sense to split the pattern navigation "functions" into
their own new concept, and maybe borrow some of the window function
infrastructure for it?

Thanks!
--Jacob

  16. Re: Row pattern recognition

    Vik Fearing <vik@postgresfriends.org> — 2023-07-21T23:38:01Z

    On 7/22/23 01:14, Jacob Champion wrote:
> On 7/20/23 17:07, Vik Fearing wrote:
>> On 7/21/23 01:36, Jacob Champion wrote:
>>> (I've attached two failing tests against v2, to hopefully better
>>> illustrate, along with what I_think_  should be the correct results.)
>>
>> Almost.  You are matching 07-01-2023 but the condition is "price > 100".
> 
> D'oh. Correction attached. I think :)

This looks correct to my human brain.  Thanks!
-- 
Vik Fearing

  17. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-07-22T01:11:49Z

    >> One small concern is how to convert pattern variables to regex
>> expression which our regex enginer understands. Suppose,
>> 
>> PATTERN UP+
>> 
>> Currently I convert "UP+" to "U+" so that it can be fed to the regexp
>> engine. In order to do that, we need to know which part of the pattern
>> (UP+) is the pattern variable ("UP"). For "UP+" it's quite easy. But
>> for more complex regular expressions it would be not, unless PATTERN
>> grammer can be analyzed by our parser to know which part is the
>> pattern variable.
> 
> Is the eventual plan to generate multiple alternatives, and run the
> regex against them one at a time?

Yes, that's my plan.

>>> I'm not familiar enough with this code yet to offer very concrete
>>> suggestions, sorry... But at some point in the future, we need to be
>>> able to skip forward and backward from arbitrary points, like
>>>
>>>     DEFINE B AS B.price > PREV(FIRST(A.price), 3)
>>>
>>> so there won't be just one pair of "previous and next" tuples.
>> 
>> Yes, I know.
> 
> I apologize. I got overexplain-y.

No problem. Thank you for reminding me it.

>>> Maybe
>>> that can help clarify the design? It feels like it'll need to eventually
>>> be a "real" function that operates on the window state, even if it
>>> doesn't support all of the possible complexities in v1.
>> 
>> Unfortunately an window function can not call other window functions.
> 
> Can that restriction be lifted for the EXPR_KIND_RPR_DEFINE case?

I am not sure at this point. Current PostgreSQL executor creates
WindowStatePerFuncData for each window function and aggregate
appearing in OVER clause. This means PREV/NEXT and other row pattern
navigation operators cannot have their own WindowStatePerFuncData if
they do not appear in OVER clauses in a query even if PREV/NEXT
etc. are defined as window function.

> Or
> does it make sense to split the pattern navigation "functions" into
> their own new concept, and maybe borrow some of the window function
> infrastructure for it?

Maybe. Suppose a window function executes row pattern matching using
price > PREV(price). The window function already receives
WindowStatePerFuncData. If we can pass the WindowStatePerFuncData to
PREV, we could let PREV do the real work (getting previous tuple).
I have not tried this yet, though.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  18. Re: Row pattern recognition

    Vik Fearing <vik@postgresfriends.org> — 2023-07-22T02:54:43Z

    On 7/22/23 03:11, Tatsuo Ishii wrote:
>>>> Maybe
>>>> that can help clarify the design? It feels like it'll need to eventually
>>>> be a "real" function that operates on the window state, even if it
>>>> doesn't support all of the possible complexities in v1.
>>> Unfortunately an window function can not call other window functions.
>> Can that restriction be lifted for the EXPR_KIND_RPR_DEFINE case?

> I am not sure at this point. Current PostgreSQL executor creates
> WindowStatePerFuncData for each window function and aggregate
> appearing in OVER clause. This means PREV/NEXT and other row pattern
> navigation operators cannot have their own WindowStatePerFuncData if
> they do not appear in OVER clauses in a query even if PREV/NEXT
> etc. are defined as window function.
> 
>> Or
>> does it make sense to split the pattern navigation "functions" into
>> their own new concept, and maybe borrow some of the window function
>> infrastructure for it?

> Maybe. Suppose a window function executes row pattern matching using
> price > PREV(price). The window function already receives
> WindowStatePerFuncData. If we can pass the WindowStatePerFuncData to
> PREV, we could let PREV do the real work (getting previous tuple).
> I have not tried this yet, though.


I don't understand this logic.  Window functions work over a window 
frame.  What we are talking about here is *defining* a window frame. 
How can a window function execute row pattern matching?
-- 
Vik Fearing

  19. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-07-22T06:14:46Z

    > On 7/22/23 03:11, Tatsuo Ishii wrote:
>>>>> Maybe
>>>>> that can help clarify the design? It feels like it'll need to
>>>>> eventually
>>>>> be a "real" function that operates on the window state, even if it
>>>>> doesn't support all of the possible complexities in v1.
>>>> Unfortunately an window function can not call other window functions.
>>> Can that restriction be lifted for the EXPR_KIND_RPR_DEFINE case?
> 
>> I am not sure at this point. Current PostgreSQL executor creates
>> WindowStatePerFuncData for each window function and aggregate
>> appearing in OVER clause. This means PREV/NEXT and other row pattern
>> navigation operators cannot have their own WindowStatePerFuncData if
>> they do not appear in OVER clauses in a query even if PREV/NEXT
>> etc. are defined as window function.
>> 
>>> Or
>>> does it make sense to split the pattern navigation "functions" into
>>> their own new concept, and maybe borrow some of the window function
>>> infrastructure for it?
> 
>> Maybe. Suppose a window function executes row pattern matching using
>> price > PREV(price). The window function already receives
>> WindowStatePerFuncData. If we can pass the WindowStatePerFuncData to
>> PREV, we could let PREV do the real work (getting previous tuple).
>> I have not tried this yet, though.
> 
> 
> I don't understand this logic.  Window functions work over a window
> frame.

Yes.

> What we are talking about here is *defining* a window
> frame.

Well, we are defining a "reduced" window frame within a (full) window
frame. A "reduced" window frame is calculated each time when a window
function is called.

> How can a window function execute row pattern matching?

A window function is called for each row fed by an outer plan. It
fetches current, previous and next row to execute pattern matching. If
it matches, the window function moves to next row and repeat the
process, until pattern match fails.

Below is an example window function to execute pattern matching (I
will include this in the v3 patch). row_is_in_reduced_frame() is a
function to execute pattern matching. It returns the number of rows in
the reduced frame if pattern match succeeds. If succeeds, the function
returns the last row in the reduced frame instead of the last row in
the full window frame.

/*
 * last_value
 * return the value of VE evaluated on the last row of the
 * window frame, per spec.
 */
Datum
window_last_value(PG_FUNCTION_ARGS)
{
	WindowObject winobj = PG_WINDOW_OBJECT();
	Datum		result;
	bool		isnull;
	int64		abspos;
	int			num_reduced_frame;

	abspos = WinGetCurrentPosition(winobj);
	num_reduced_frame = row_is_in_reduced_frame(winobj, abspos);

	if (num_reduced_frame == 0)
		/* no RPR is involved */
		result = WinGetFuncArgInFrame(winobj, 0,
									  0, WINDOW_SEEK_TAIL, true,
									  &isnull, NULL);
	else if (num_reduced_frame > 0)
		/* get last row value in the reduced frame */
		result = WinGetFuncArgInFrame(winobj, 0,
									  num_reduced_frame - 1, WINDOW_SEEK_HEAD, true,
									  &isnull, NULL);
	else
		/* RPR is involved and current row is unmatched or skipped */
		isnull = true;

	if (isnull)
		PG_RETURN_NULL();

	PG_RETURN_DATUM(result);
}

  20. Re: Row pattern recognition

    Vik Fearing <vik@postgresfriends.org> — 2023-07-23T21:29:46Z

    On 7/22/23 08:14, Tatsuo Ishii wrote:
>> On 7/22/23 03:11, Tatsuo Ishii wrote:
>>> Maybe. Suppose a window function executes row pattern matching using
>>> price > PREV(price). The window function already receives
>>> WindowStatePerFuncData. If we can pass the WindowStatePerFuncData to
>>> PREV, we could let PREV do the real work (getting previous tuple).
>>> I have not tried this yet, though.
>>
>> I don't understand this logic.  Window functions work over a window
>> frame.
> 
> Yes.
> 
>> What we are talking about here is *defining* a window
>> frame.
> 
> Well, we are defining a "reduced" window frame within a (full) window
> frame. A "reduced" window frame is calculated each time when a window
> function is called.


Why?  It should only be recalculated when the current row changes and we 
need a new frame.  The reduced window frame *is* the window frame for 
all functions over that window.


>> How can a window function execute row pattern matching?
> 
> A window function is called for each row fed by an outer plan. It
> fetches current, previous and next row to execute pattern matching. If
> it matches, the window function moves to next row and repeat the
> process, until pattern match fails.
> 
> Below is an example window function to execute pattern matching (I
> will include this in the v3 patch). row_is_in_reduced_frame() is a
> function to execute pattern matching. It returns the number of rows in
> the reduced frame if pattern match succeeds. If succeeds, the function
> returns the last row in the reduced frame instead of the last row in
> the full window frame.


I strongly disagree with this.  Window function do not need to know how 
the frame is defined, and indeed they should not.  WinGetFuncArgInFrame 
should answer yes or no and the window function just works on that. 
Otherwise we will get extension (and possibly even core) functions that 
don't handle the frame properly.
-- 
Vik Fearing

  21. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-07-24T00:22:40Z

    >>> What we are talking about here is *defining* a window
>>> frame.
>> Well, we are defining a "reduced" window frame within a (full) window
>> frame. A "reduced" window frame is calculated each time when a window
>> function is called.
> 
> 
> Why?  It should only be recalculated when the current row changes and
> we need a new frame.  The reduced window frame *is* the window frame
> for all functions over that window.

We already recalculate a frame each time a row is processed even
without RPR. See ExecWindowAgg.

Also RPR always requires a frame option ROWS BETWEEN CURRENT ROW,
which means the frame head is changed each time current row position
changes.

> I strongly disagree with this.  Window function do not need to know
> how the frame is defined, and indeed they should not.

We already break the rule by defining *support functions. See
windowfuncs.c.

> WinGetFuncArgInFrame should answer yes or no and the window function
> just works on that. Otherwise we will get extension (and possibly even
> core) functions that don't handle the frame properly.

Maybe I can move row_is_in_reduced_frame into WinGetFuncArgInFrame
just for convenience.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  22. Re: Row pattern recognition

    Vik Fearing <vik@postgresfriends.org> — 2023-07-24T23:14:37Z

    On 7/24/23 02:22, Tatsuo Ishii wrote:
>>>> What we are talking about here is *defining* a window
>>>> frame.
>>> Well, we are defining a "reduced" window frame within a (full) window
>>> frame. A "reduced" window frame is calculated each time when a window
>>> function is called.
>>
>>
>> Why?  It should only be recalculated when the current row changes and
>> we need a new frame.  The reduced window frame *is* the window frame
>> for all functions over that window.
> 
> We already recalculate a frame each time a row is processed even
> without RPR. See ExecWindowAgg.

Yes, after each row.  Not for each function.

> Also RPR always requires a frame option ROWS BETWEEN CURRENT ROW,
> which means the frame head is changed each time current row position
> changes.

Off topic for now: I wonder why this restriction is in place and whether 
we should respect or ignore it.  That is a discussion for another time, 
though.

>> I strongly disagree with this.  Window function do not need to know
>> how the frame is defined, and indeed they should not.
> 
> We already break the rule by defining *support functions. See
> windowfuncs.c.

The support functions don't know anything about the frame, they just 
know when a window function is monotonically increasing and execution 
can either stop or be "passed through".

>> WinGetFuncArgInFrame should answer yes or no and the window function
>> just works on that. Otherwise we will get extension (and possibly even
>> core) functions that don't handle the frame properly.
> 
> Maybe I can move row_is_in_reduced_frame into WinGetFuncArgInFrame
> just for convenience.

I have two comments about this:

It isn't just for convenience, it is for correctness.  The window 
functions do not need to know which rows they are *not* operating on.

There is no such thing as a "full" or "reduced" frame.  The standard 
uses those terms to explain the difference between before and after RPR 
is applied, but window functions do not get to choose which frame they 
apply over.  They only ever apply over the reduced window frame.
-- 
Vik Fearing

  23. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-07-25T12:35:04Z

    Hi,

> diff --git a/src/test/regress/expected/rpr.out b/src/test/regress/expected/rpr.out
> index 6bf8818911..f3fd22de2a 100644
> --- a/src/test/regress/expected/rpr.out
> +++ b/src/test/regress/expected/rpr.out
> @@ -230,6 +230,79 @@ SELECT company, tdate, price, rpr(price) OVER w FROM stock
>   company2 | 07-10-2023 |  1300 |     
>  (20 rows)
>  
> +-- match everything
> +SELECT company, tdate, price, rpr(price) OVER w FROM stock
> + WINDOW w AS (
> + PARTITION BY company
> + ORDER BY tdate
> + ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
> + AFTER MATCH SKIP TO NEXT ROW

It seems it's a result with AFTER MATCH SKIP PAST LAST ROW.

> + INITIAL
> + PATTERN (A+)
> + DEFINE
> +  A AS TRUE
> +);
> + company  |   tdate    | price | rpr 
> +----------+------------+-------+-----
> + company1 | 07-01-2023 |   100 | 100
> + company1 | 07-02-2023 |   200 |    
> + company1 | 07-03-2023 |   150 |    
> + company1 | 07-04-2023 |   140 |    
> + company1 | 07-05-2023 |   150 |    
> + company1 | 07-06-2023 |    90 |    
> + company1 | 07-07-2023 |   110 |    
> + company1 | 07-08-2023 |   130 |    
> + company1 | 07-09-2023 |   120 |    
> + company1 | 07-10-2023 |   130 |    
> + company2 | 07-01-2023 |    50 |  50
> + company2 | 07-02-2023 |  2000 |    
> + company2 | 07-03-2023 |  1500 |    
> + company2 | 07-04-2023 |  1400 |    
> + company2 | 07-05-2023 |  1500 |    
> + company2 | 07-06-2023 |    60 |    
> + company2 | 07-07-2023 |  1100 |    
> + company2 | 07-08-2023 |  1300 |    
> + company2 | 07-09-2023 |  1200 |    
> + company2 | 07-10-2023 |  1300 |    
> +(20 rows)
> +
> +-- backtracking with reclassification of rows
> +SELECT company, tdate, price, rpr(price) OVER w FROM stock
> + WINDOW w AS (
> + PARTITION BY company
> + ORDER BY tdate
> + ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
> + AFTER MATCH SKIP TO NEXT ROW
> + INITIAL
> + PATTERN (A+ B+)
> + DEFINE
> +  A AS price > 100,
> +  B AS price > 100
> +);
> + company  |   tdate    | price | rpr  
> +----------+------------+-------+------
> + company1 | 07-01-2023 |   100 |     
> + company1 | 07-02-2023 |   200 |  200
> + company1 | 07-03-2023 |   150 |     
> + company1 | 07-04-2023 |   140 |     
> + company1 | 07-05-2023 |   150 |     
> + company1 | 07-06-2023 |    90 |     
> + company1 | 07-07-2023 |   110 |  110
> + company1 | 07-08-2023 |   130 |     
> + company1 | 07-09-2023 |   120 |     
> + company1 | 07-10-2023 |   130 |     
> + company2 | 07-01-2023 |    50 |     
> + company2 | 07-02-2023 |  2000 | 2000
> + company2 | 07-03-2023 |  1500 |     
> + company2 | 07-04-2023 |  1400 |     
> + company2 | 07-05-2023 |  1500 |     
> + company2 | 07-06-2023 |    60 |     
> + company2 | 07-07-2023 |  1100 | 1100
> + company2 | 07-08-2023 |  1300 |     
> + company2 | 07-09-2023 |  1200 |     
> + company2 | 07-10-2023 |  1300 |     
> +(20 rows)
> +
>  --
>  -- Error cases
>  --
> diff --git a/src/test/regress/sql/rpr.sql b/src/test/regress/sql/rpr.sql
> index 951c9abfe9..f1cd0369f4 100644
> --- a/src/test/regress/sql/rpr.sql
> +++ b/src/test/regress/sql/rpr.sql
> @@ -94,6 +94,33 @@ SELECT company, tdate, price, rpr(price) OVER w FROM stock
>    UPDOWN AS price > PREV(price) AND price > NEXT(price)
>  );
>  
> +-- match everything
> +SELECT company, tdate, price, rpr(price) OVER w FROM stock
> + WINDOW w AS (
> + PARTITION BY company
> + ORDER BY tdate
> + ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
> + AFTER MATCH SKIP TO NEXT ROW
> + INITIAL
> + PATTERN (A+)
> + DEFINE
> +  A AS TRUE
> +);
> +
> +-- backtracking with reclassification of rows
> +SELECT company, tdate, price, rpr(price) OVER w FROM stock
> + WINDOW w AS (
> + PARTITION BY company
> + ORDER BY tdate
> + ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
> + AFTER MATCH SKIP TO NEXT ROW
> + INITIAL
> + PATTERN (A+ B+)
> + DEFINE
> +  A AS price > 100,
> +  B AS price > 100
> +);
> +
>  --
>  -- Error cases
>  --

  24. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-07-26T12:21:34Z

    Attached is the v3 patch. In this patch following changes are made.

(1) I completely changed the pattern matching engine so that it
performs backtracking. Now the engine evaluates all pattern elements
defined in PATTER against each row, saving matched pattern variables
in a string per row. For example if the pattern element A and B
evaluated to true, a string "AB" is created for current row.

This continues until all pattern matching fails or encounters the end
of full window frame/partition. After that, the pattern matching
engine creates all possible "pattern strings" and apply the regular
expression matching to each. For example if we have row 0 = "AB" row 1
= "C", possible pattern strings are: "AC" and "BC".

If it matches, the length of matching substring is saved. After all
possible trials are done, the longest matching substring is chosen and
it becomes the width (number of rows) in the reduced window frame.

See row_is_in_reduced_frame, search_str_set and search_str_set_recurse
in nodeWindowAggs.c for more details. For now I use a naive depth
first search and probably there is a lot of rooms for optimization
(for example rewriting it without using
recursion). Suggestions/patches are welcome.

Jacob Champion wrote:
> It looks like the + qualifier has trouble when it meets the end of the
> frame. For instance, try removing the last row of the 'stock' table in
> rpr.sql; some of the final matches will disappear unexpectedly. Or try a
> pattern like
> 
>     PATTERN ( a+ )
>      DEFINE a AS TRUE
> 
> which doesn't seem to match anything in my testing.
> 
> There's also the issue of backtracking in the face of reclassification,
> as I think Vik was alluding to upthread. The pattern
> 
>     PATTERN ( a+ b+ )
>      DEFINE a AS col = 2,
>             b AS col = 2

With the new engine, cases above do not fail anymore. See new
regression test cases. Thanks for providing valuable test cases!

(2) Make window functions RPR aware. Now first_value, last_value, and
nth_value recognize RPR (maybe first_value do not need any change?)

Vik Fearing wrote:
> I strongly disagree with this.  Window function do not need to know
> how the frame is defined, and indeed they should not.
> WinGetFuncArgInFrame should answer yes or no and the window function
> just works on that. Otherwise we will get extension (and possibly even
> core) functions that don't handle the frame properly.

So I moved row_is_in_reduce_frame into WinGetFuncArgInFrame so that
those window functions are not needed to be changed.

(3) Window function rpr was removed. We can use first_value instead.

(4) Remaining tasks/issues.

- For now I disable WinSetMarkPosition because RPR often needs to
  access a row before the mark is set. We need to fix this in the
  future.

- I am working on making window aggregates RPR aware now. The
  implementation is in progress and far from completeness. An example
  is below. I think row 2, 3, 4 of "count" column should be NULL
  instead of 3, 2, 0, 0. Same thing can be said to other
  rows. Probably this is an effect of moving aggregate but I still
  studying the window aggregation code.

SELECT company, tdate, first_value(price) OVER W, count(*) OVER w FROM stock
 WINDOW w AS (
 PARTITION BY company
 ORDER BY tdate
 ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
 AFTER MATCH SKIP PAST LAST ROW
 INITIAL
 PATTERN (START UP+ DOWN+)
 DEFINE
  START AS TRUE,
  UP AS price > PREV(price),
  DOWN AS price < PREV(price)
);
 company  |   tdate    | first_value | count 
----------+------------+-------------+-------
 company1 | 2023-07-01 |         100 |     4
 company1 | 2023-07-02 |             |     3
 company1 | 2023-07-03 |             |     2
 company1 | 2023-07-04 |             |     0
 company1 | 2023-07-05 |             |     0
 company1 | 2023-07-06 |          90 |     4
 company1 | 2023-07-07 |             |     3
 company1 | 2023-07-08 |             |     2
 company1 | 2023-07-09 |             |     0
 company1 | 2023-07-10 |             |     0
 company2 | 2023-07-01 |          50 |     4
 company2 | 2023-07-02 |             |     3
 company2 | 2023-07-03 |             |     2
 company2 | 2023-07-04 |             |     0
 company2 | 2023-07-05 |             |     0
 company2 | 2023-07-06 |          60 |     4
 company2 | 2023-07-07 |             |     3
 company2 | 2023-07-08 |             |     2
 company2 | 2023-07-09 |             |     0
 company2 | 2023-07-10 |             |     0

- If attributes appearing in DEFINE are not used in the target list, query fails.

SELECT company, tdate, count(*) OVER w FROM stock
 WINDOW w AS (
 PARTITION BY company
 ORDER BY tdate
 ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
 AFTER MATCH SKIP PAST LAST ROW
 INITIAL
 PATTERN (START UP+ DOWN+)
 DEFINE
  START AS TRUE,
  UP AS price > PREV(price),
  DOWN AS price < PREV(price)
);
ERROR:  attribute number 3 exceeds number of columns 2

This is because attributes appearing in DEFINE are not added to the
target list. I am looking for way to teach planner to add attributes
appearing in DEFINE.

I am going to add this thread to CommitFest and plan to add both of
you as reviewers. Thanks in advance.
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  25. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-07-26T20:22:30Z

    > I am going to add this thread to CommitFest and plan to add both of
> you as reviewers. Thanks in advance.

Done.
https://commitfest.postgresql.org/44/4460/

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  26. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-07-28T07:09:53Z

    >> We already recalculate a frame each time a row is processed even
>> without RPR. See ExecWindowAgg.
> 
> Yes, after each row.  Not for each function.

Ok, I understand now. Closer look at the code, I realized that each
window function calls update_frameheadpos, which computes the frame
head position. But actually it checks winstate->framehead_valid and if
it's already true (probably by other window function), then it does
nothing.

>> Also RPR always requires a frame option ROWS BETWEEN CURRENT ROW,
>> which means the frame head is changed each time current row position
>> changes.
> 
> Off topic for now: I wonder why this restriction is in place and
> whether we should respect or ignore it.  That is a discussion for
> another time, though.

My guess is, it is because other than ROWS BETWEEN CURRENT ROW has
little or no meaning. Consider following example:

SELECT i, first_value(i) OVER w
  FROM (VALUES (1), (2), (3), (4)) AS v (i)
  WINDOW w AS (
   ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
   AFTER MATCH SKIP PAST LAST ROW
   PATTERN (A)
   DEFINE
    A AS i = 1 OR i = 3
);

In this example ROWS BETWEEN CURRENT ROW gives frames with i = 1 and i
= 3.

 i | first_value 
---+-------------
 1 |           1
 2 |            
 3 |           3
 4 |            
(4 rows)

But what would happen with ROWS BETWEEN UNBOUNDED PRECEDING AND
UNBOUNDED FOLLOWING?  Probably the frame i = 3 will be missed as
at i = 2, PATTERN is not satisfied and compution of the reduced frame
stops.

 i | first_value 
---+-------------
 1 |           1
 2 |            
 3 |           
 4 |            
(4 rows)

This is not very useful for users.

>>> I strongly disagree with this.  Window function do not need to know
>>> how the frame is defined, and indeed they should not.
>> We already break the rule by defining *support functions. See
>> windowfuncs.c.
> The support functions don't know anything about the frame, they just
> know when a window function is monotonically increasing and execution
> can either stop or be "passed through".

I see following code in window_row_number_support:

		/*
		 * The frame options can always become "ROWS BETWEEN UNBOUNDED
		 * PRECEDING AND CURRENT ROW".  row_number() always just increments by
		 * 1 with each row in the partition.  Using ROWS instead of RANGE
		 * saves effort checking peer rows during execution.
		 */
		req->frameOptions = (FRAMEOPTION_NONDEFAULT |
							 FRAMEOPTION_ROWS |
							 FRAMEOPTION_START_UNBOUNDED_PRECEDING |
							 FRAMEOPTION_END_CURRENT_ROW);

I think it not only knows about frame but it even changes the frame
options. This seems far from "don't know anything about the frame", no?

> I have two comments about this:
> 
> It isn't just for convenience, it is for correctness.  The window
> functions do not need to know which rows they are *not* operating on.
> 
> There is no such thing as a "full" or "reduced" frame.  The standard
> uses those terms to explain the difference between before and after
> RPR is applied, but window functions do not get to choose which frame
> they apply over.  They only ever apply over the reduced window frame.

I agree that "full window frame" and "reduced window frame" do not
exist at the same time, and in the end (after computation of reduced
frame), only "reduced" frame is visible to window
functions/aggregates. But I still do think that "full window frame"
and "reduced window frame" are important concept to explain/understand
how PRP works.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  27. Re: Row pattern recognition

    Vik Fearing <vik@postgresfriends.org> — 2023-07-28T08:56:26Z

    On 7/28/23 09:09, Tatsuo Ishii wrote:
>>> We already recalculate a frame each time a row is processed even
>>> without RPR. See ExecWindowAgg.
>>
>> Yes, after each row.  Not for each function.
> 
> Ok, I understand now. Closer look at the code, I realized that each
> window function calls update_frameheadpos, which computes the frame
> head position. But actually it checks winstate->framehead_valid and if
> it's already true (probably by other window function), then it does
> nothing.
> 
>>> Also RPR always requires a frame option ROWS BETWEEN CURRENT ROW,
>>> which means the frame head is changed each time current row position
>>> changes.
>>
>> Off topic for now: I wonder why this restriction is in place and
>> whether we should respect or ignore it.  That is a discussion for
>> another time, though.
> 
> My guess is, it is because other than ROWS BETWEEN CURRENT ROW has
> little or no meaning. Consider following example:

Yes, that makes sense.

>>>> I strongly disagree with this.  Window function do not need to know
>>>> how the frame is defined, and indeed they should not.
>>> We already break the rule by defining *support functions. See
>>> windowfuncs.c.
>> The support functions don't know anything about the frame, they just
>> know when a window function is monotonically increasing and execution
>> can either stop or be "passed through".
> 
> I see following code in window_row_number_support:
> 
> 		/*
> 		 * The frame options can always become "ROWS BETWEEN UNBOUNDED
> 		 * PRECEDING AND CURRENT ROW".  row_number() always just increments by
> 		 * 1 with each row in the partition.  Using ROWS instead of RANGE
> 		 * saves effort checking peer rows during execution.
> 		 */
> 		req->frameOptions = (FRAMEOPTION_NONDEFAULT |
> 							 FRAMEOPTION_ROWS |
> 							 FRAMEOPTION_START_UNBOUNDED_PRECEDING |
> 							 FRAMEOPTION_END_CURRENT_ROW);
> 
> I think it not only knows about frame but it even changes the frame
> options. This seems far from "don't know anything about the frame", no?

That's the planner support function.  The row_number() function itself 
is not even allowed to *have* a frame, per spec.  We allow it, but as 
you can see from that support function, we completely replace it.

So all of the partition-level window functions are not affected by RPR 
anyway.

>> I have two comments about this:
>>
>> It isn't just for convenience, it is for correctness.  The window
>> functions do not need to know which rows they are *not* operating on.
>>
>> There is no such thing as a "full" or "reduced" frame.  The standard
>> uses those terms to explain the difference between before and after
>> RPR is applied, but window functions do not get to choose which frame
>> they apply over.  They only ever apply over the reduced window frame.
> 
> I agree that "full window frame" and "reduced window frame" do not
> exist at the same time, and in the end (after computation of reduced
> frame), only "reduced" frame is visible to window
> functions/aggregates. But I still do think that "full window frame"
> and "reduced window frame" are important concept to explain/understand
> how PRP works.

If we are just using those terms for documentation, then okay.
-- 
Vik Fearing

  28. Re: Row pattern recognition

    Vik Fearing <vik@postgresfriends.org> — 2023-07-28T09:21:25Z

    On 7/26/23 14:21, Tatsuo Ishii wrote:
> Attached is the v3 patch. In this patch following changes are made.

Excellent.  Thanks!

A few quick comments:

- PERMUTE is still misspelled as PREMUTE

- PATTERN variables do not have to exist in the DEFINE clause.  They are 
considered TRUE if not present.

> (1) I completely changed the pattern matching engine so that it
> performs backtracking. Now the engine evaluates all pattern elements
> defined in PATTER against each row, saving matched pattern variables
> in a string per row. For example if the pattern element A and B
> evaluated to true, a string "AB" is created for current row.
> 
> This continues until all pattern matching fails or encounters the end
> of full window frame/partition. After that, the pattern matching
> engine creates all possible "pattern strings" and apply the regular
> expression matching to each. For example if we have row 0 = "AB" row 1
> = "C", possible pattern strings are: "AC" and "BC".
> 
> If it matches, the length of matching substring is saved. After all
> possible trials are done, the longest matching substring is chosen and
> it becomes the width (number of rows) in the reduced window frame.
> 
> See row_is_in_reduced_frame, search_str_set and search_str_set_recurse
> in nodeWindowAggs.c for more details. For now I use a naive depth
> first search and probably there is a lot of rooms for optimization
> (for example rewriting it without using
> recursion). Suggestions/patches are welcome.

My own reviews will only focus on correctness for now.  Once we get a 
good set of regression tests all passing, I will focus more on 
optimization.  Of course, others might want to review the performance now.

> Vik Fearing wrote:
>> I strongly disagree with this.  Window function do not need to know
>> how the frame is defined, and indeed they should not.
>> WinGetFuncArgInFrame should answer yes or no and the window function
>> just works on that. Otherwise we will get extension (and possibly even
>> core) functions that don't handle the frame properly.
> 
> So I moved row_is_in_reduce_frame into WinGetFuncArgInFrame so that
> those window functions are not needed to be changed.
> 
> (3) Window function rpr was removed. We can use first_value instead.

Excellent.

> (4) Remaining tasks/issues.
> 
> - For now I disable WinSetMarkPosition because RPR often needs to
>    access a row before the mark is set. We need to fix this in the
>    future.

Noted, and agreed.

> - I am working on making window aggregates RPR aware now. The
>    implementation is in progress and far from completeness. An example
>    is below. I think row 2, 3, 4 of "count" column should be NULL
>    instead of 3, 2, 0, 0. Same thing can be said to other
>    rows. Probably this is an effect of moving aggregate but I still
>    studying the window aggregation code.

This tells me again that RPR is not being run in the right place.  All 
windowed aggregates and frame-level window functions should Just Work 
with no modification.

> SELECT company, tdate, first_value(price) OVER W, count(*) OVER w FROM stock
>   WINDOW w AS (
>   PARTITION BY company
>   ORDER BY tdate
>   ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>   AFTER MATCH SKIP PAST LAST ROW
>   INITIAL
>   PATTERN (START UP+ DOWN+)
>   DEFINE
>    START AS TRUE,
>    UP AS price > PREV(price),
>    DOWN AS price < PREV(price)
> );
>   company  |   tdate    | first_value | count
> ----------+------------+-------------+-------
>   company1 | 2023-07-01 |         100 |     4
>   company1 | 2023-07-02 |             |     3
>   company1 | 2023-07-03 |             |     2
>   company1 | 2023-07-04 |             |     0
>   company1 | 2023-07-05 |             |     0
>   company1 | 2023-07-06 |          90 |     4
>   company1 | 2023-07-07 |             |     3
>   company1 | 2023-07-08 |             |     2
>   company1 | 2023-07-09 |             |     0
>   company1 | 2023-07-10 |             |     0
>   company2 | 2023-07-01 |          50 |     4
>   company2 | 2023-07-02 |             |     3
>   company2 | 2023-07-03 |             |     2
>   company2 | 2023-07-04 |             |     0
>   company2 | 2023-07-05 |             |     0
>   company2 | 2023-07-06 |          60 |     4
>   company2 | 2023-07-07 |             |     3
>   company2 | 2023-07-08 |             |     2
>   company2 | 2023-07-09 |             |     0
>   company2 | 2023-07-10 |             |     0

In this scenario, row 1's frame is the first 5 rows and specified SKIP 
PAST LAST ROW, so rows 2-5 don't have *any* frame (because they are 
skipped) and the result of the outer count should be 0 for all of them 
because there are no rows in the frame.

When we get to adding count in the MEASURES clause, there will be a 
difference between no match and empty match, but that does not apply here.

> I am going to add this thread to CommitFest and plan to add both of
> you as reviewers. Thanks in advance.

My pleasure.  Thank you for working on this difficult feature.
-- 
Vik Fearing

  29. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-07-28T11:02:30Z

    >> Attached is the v3 patch. In this patch following changes are made.
> 
> Excellent.  Thanks!

You are welcome!

> A few quick comments:
> 
> - PERMUTE is still misspelled as PREMUTE

Oops. Will fix.

> - PATTERN variables do not have to exist in the DEFINE clause.  They are
> - considered TRUE if not present.

Do you think we really need this? I found a criticism regarding this.

https://link.springer.com/article/10.1007/s13222-022-00404-3
"3.2 Explicit Definition of All Row Pattern Variables"

What do you think?

>> - I am working on making window aggregates RPR aware now. The
>>    implementation is in progress and far from completeness. An example
>>    is below. I think row 2, 3, 4 of "count" column should be NULL
>>    instead of 3, 2, 0, 0. Same thing can be said to other
>>    rows. Probably this is an effect of moving aggregate but I still
>>    studying the window aggregation code.
> 
> This tells me again that RPR is not being run in the right place.  All
> windowed aggregates and frame-level window functions should Just Work
> with no modification.

I am not touching each aggregate function. I am modifying
eval_windowaggregates() in nodeWindowAgg.c, which calls each aggregate
function. Do you think it's not the right place to make window
aggregates RPR aware?

>> SELECT company, tdate, first_value(price) OVER W, count(*) OVER w FROM
>> stock
>>   WINDOW w AS (
>>   PARTITION BY company
>>   ORDER BY tdate
>>   ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>>   AFTER MATCH SKIP PAST LAST ROW
>>   INITIAL
>>   PATTERN (START UP+ DOWN+)
>>   DEFINE
>>    START AS TRUE,
>>    UP AS price > PREV(price),
>>    DOWN AS price < PREV(price)
>> );
>>   company  |   tdate    | first_value | count
>> ----------+------------+-------------+-------
>>   company1 | 2023-07-01 |         100 |     4
>>   company1 | 2023-07-02 |             |     3
>>   company1 | 2023-07-03 |             |     2
>>   company1 | 2023-07-04 |             |     0
>>   company1 | 2023-07-05 |             |     0
>>   company1 | 2023-07-06 |          90 |     4
>>   company1 | 2023-07-07 |             |     3
>>   company1 | 2023-07-08 |             |     2
>>   company1 | 2023-07-09 |             |     0
>>   company1 | 2023-07-10 |             |     0
>>   company2 | 2023-07-01 |          50 |     4
>>   company2 | 2023-07-02 |             |     3
>>   company2 | 2023-07-03 |             |     2
>>   company2 | 2023-07-04 |             |     0
>>   company2 | 2023-07-05 |             |     0
>>   company2 | 2023-07-06 |          60 |     4
>>   company2 | 2023-07-07 |             |     3
>>   company2 | 2023-07-08 |             |     2
>>   company2 | 2023-07-09 |             |     0
>>   company2 | 2023-07-10 |             |     0
> 
> In this scenario, row 1's frame is the first 5 rows and specified SKIP
> PAST LAST ROW, so rows 2-5 don't have *any* frame (because they are
> skipped) and the result of the outer count should be 0 for all of them
> because there are no rows in the frame.

Ok. Just I want to make sure. If it's other aggregates like sum or
avg, the result of the outer aggregates should be NULL.

> When we get to adding count in the MEASURES clause, there will be a
> difference between no match and empty match, but that does not apply
> here.

Can you elaborate more? I understand that "no match" and "empty match"
are different things. But I do not understand how the difference
affects the result of count.

>> I am going to add this thread to CommitFest and plan to add both of
>> you as reviewers. Thanks in advance.
> 
> My pleasure.  Thank you for working on this difficult feature.

Thank you for accepting being registered as a reviewer. Your comments
are really helpful.
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  30. Re: Row pattern recognition

    Vik Fearing <vik@postgresfriends.org> — 2023-07-28T12:36:58Z

    On 7/28/23 13:02, Tatsuo Ishii wrote:
>>> Attached is the v3 patch. In this patch following changes are made.
>>
>> - PATTERN variables do not have to exist in the DEFINE clause.  They are
>> - considered TRUE if not present.
> 
> Do you think we really need this? I found a criticism regarding this.
> 
> https://link.springer.com/article/10.1007/s13222-022-00404-3
> "3.2 Explicit Definition of All Row Pattern Variables"
> 
> What do you think?

I think that a large part of obeying the standard is to allow queries 
from other engines to run the same on ours.  The standard does not 
require the pattern variables to be defined and so there are certainly 
queries out there without them, and that hurts migrating to PostgreSQL.

>>> - I am working on making window aggregates RPR aware now. The
>>>     implementation is in progress and far from completeness. An example
>>>     is below. I think row 2, 3, 4 of "count" column should be NULL
>>>     instead of 3, 2, 0, 0. Same thing can be said to other
>>>     rows. Probably this is an effect of moving aggregate but I still
>>>     studying the window aggregation code.
>>
>> This tells me again that RPR is not being run in the right place.  All
>> windowed aggregates and frame-level window functions should Just Work
>> with no modification.
> 
> I am not touching each aggregate function. I am modifying
> eval_windowaggregates() in nodeWindowAgg.c, which calls each aggregate
> function. Do you think it's not the right place to make window
> aggregates RPR aware?

Oh, okay.

>>> SELECT company, tdate, first_value(price) OVER W, count(*) OVER w FROM
>>> stock
>>>    WINDOW w AS (
>>>    PARTITION BY company
>>>    ORDER BY tdate
>>>    ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>>>    AFTER MATCH SKIP PAST LAST ROW
>>>    INITIAL
>>>    PATTERN (START UP+ DOWN+)
>>>    DEFINE
>>>     START AS TRUE,
>>>     UP AS price > PREV(price),
>>>     DOWN AS price < PREV(price)
>>> );
>>>    company  |   tdate    | first_value | count
>>> ----------+------------+-------------+-------
>>>    company1 | 2023-07-01 |         100 |     4
>>>    company1 | 2023-07-02 |             |     3
>>>    company1 | 2023-07-03 |             |     2
>>>    company1 | 2023-07-04 |             |     0
>>>    company1 | 2023-07-05 |             |     0
>>>    company1 | 2023-07-06 |          90 |     4
>>>    company1 | 2023-07-07 |             |     3
>>>    company1 | 2023-07-08 |             |     2
>>>    company1 | 2023-07-09 |             |     0
>>>    company1 | 2023-07-10 |             |     0
>>>    company2 | 2023-07-01 |          50 |     4
>>>    company2 | 2023-07-02 |             |     3
>>>    company2 | 2023-07-03 |             |     2
>>>    company2 | 2023-07-04 |             |     0
>>>    company2 | 2023-07-05 |             |     0
>>>    company2 | 2023-07-06 |          60 |     4
>>>    company2 | 2023-07-07 |             |     3
>>>    company2 | 2023-07-08 |             |     2
>>>    company2 | 2023-07-09 |             |     0
>>>    company2 | 2023-07-10 |             |     0
>>
>> In this scenario, row 1's frame is the first 5 rows and specified SKIP
>> PAST LAST ROW, so rows 2-5 don't have *any* frame (because they are
>> skipped) and the result of the outer count should be 0 for all of them
>> because there are no rows in the frame.
> 
> Ok. Just I want to make sure. If it's other aggregates like sum or
> avg, the result of the outer aggregates should be NULL.

They all behave the same way as in a normal query when they receive no 
rows as input.

>> When we get to adding count in the MEASURES clause, there will be a
>> difference between no match and empty match, but that does not apply
>> here.
> 
> Can you elaborate more? I understand that "no match" and "empty match"
> are different things. But I do not understand how the difference
> affects the result of count.

This query:

SELECT v.a, wcnt OVER w, count(*) OVER w
FROM (VALUES ('A')) AS v (a)
WINDOW w AS (
   ORDER BY v.a
   MEASURES count(*) AS wcnt
   ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
   PATTERN (B)
   DEFINE B AS B.a = 'B'
)

produces this result:

  a | wcnt | count
---+------+-------
  A |      |     0
(1 row)

Inside the window specification, *no match* was found and so all of the 
MEASURES are null.  The count(*) in the target list however, still 
exists and operates over zero rows.

This very similar query:

SELECT v.a, wcnt OVER w, count(*) OVER w
FROM (VALUES ('A')) AS v (a)
WINDOW w AS (
   ORDER BY v.a
   MEASURES count(*) AS wcnt
   ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
   PATTERN (B?)
   DEFINE B AS B.a = 'B'
)

produces this result:

  a | wcnt | count
---+------+-------
  A |    0 |     0
(1 row)

In this case, the pattern is B? instead of just B, which produces an 
*empty match* for the MEASURES to be applied over.
-- 
Vik Fearing

  31. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-07-29T03:05:08Z

    >>> - PATTERN variables do not have to exist in the DEFINE clause.  They are
>>> - considered TRUE if not present.
>> Do you think we really need this? I found a criticism regarding this.
>> https://link.springer.com/article/10.1007/s13222-022-00404-3
>> "3.2 Explicit Definition of All Row Pattern Variables"
>> What do you think?
> 
> I think that a large part of obeying the standard is to allow queries
> from other engines to run the same on ours.  The standard does not
> require the pattern variables to be defined and so there are certainly
> queries out there without them, and that hurts migrating to
> PostgreSQL.

Yeah, migration is good point. I agree we should have the feature.

>>> When we get to adding count in the MEASURES clause, there will be a
>>> difference between no match and empty match, but that does not apply
>>> here.
>> Can you elaborate more? I understand that "no match" and "empty match"
>> are different things. But I do not understand how the difference
>> affects the result of count.
> 
> This query:
> 
> SELECT v.a, wcnt OVER w, count(*) OVER w
> FROM (VALUES ('A')) AS v (a)
> WINDOW w AS (
>   ORDER BY v.a
>   MEASURES count(*) AS wcnt
>   ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>   PATTERN (B)
>   DEFINE B AS B.a = 'B'
> )
> 
> produces this result:
> 
>  a | wcnt | count
> ---+------+-------
>  A |      |     0
> (1 row)
> 
> Inside the window specification, *no match* was found and so all of
> the MEASURES are null.  The count(*) in the target list however, still
> exists and operates over zero rows.
> 
> This very similar query:
> 
> SELECT v.a, wcnt OVER w, count(*) OVER w
> FROM (VALUES ('A')) AS v (a)
> WINDOW w AS (
>   ORDER BY v.a
>   MEASURES count(*) AS wcnt
>   ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>   PATTERN (B?)
>   DEFINE B AS B.a = 'B'
> )
> 
> produces this result:
> 
>  a | wcnt | count
> ---+------+-------
>  A |    0 |     0
> (1 row)
> 
> In this case, the pattern is B? instead of just B, which produces an
> *empty match* for the MEASURES to be applied over.

Thank you for the detailed explanation. I think I understand now.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  32. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-08-09T08:41:12Z

    Attached is the v4 patch. Differences from previous patch include:

> - PERMUTE is still misspelled as PREMUTE

Fixed.

> - PATTERN variables do not have to exist in the DEFINE clause.  They are
> - considered TRUE if not present.

Fixed. Moreover new regression test case is added.

- It was possible that tle nodes in DEFINE clause do not appear in the
  plan's target list. This makes impossible to evaluate expressions in
  the DEFINE because it does not appear in the outer plan's target
  list. To fix this, call findTargetlistEntrySQL99 (with resjunk is
  true) so that the missing TargetEntry is added to the outer plan
  later on.

- I eliminated some hacks in handling the Var node in DEFINE
  clause. Previously I replaced varattno of Var node in a plan tree by
  hand so that it refers to correct varattno in the outer plan
  node. In this patch I modified set_upper_references so that it calls
  fix_upper_expr for those Var nodes in the DEFINE clause. See v4-0003
  patch for more details.

- I found a bug with pattern matching code. It creates a string for
  subsequent regular expression matching. It uses the initial letter
  of each define variable name. For example, if the varname is "foo",
  then "f" is used. Obviously this makes trouble if we have two or
  more variables starting with same "f" (e.g. "food"). To fix this, I
  assign [a-z] to each variable instead of its initial letter. However
  this way limits us not to have more than 26 variables. I hope 26 is
  enough for most use cases.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  33. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-09-02T06:52:35Z

    Attached is the v5 patch. Differences from previous patch include:

* Resolve complaint from "PostgreSQL Patch Tester"
  https://commitfest.postgresql.org/44/4460/

- Change gram.y to use PATTERN_P instead of PATTERN. Using PATTERN seems
  to make trouble with Visual Studio build.

:
:
[10:07:57.853] FAILED: src/backend/parser/parser.a.p/meson-generated_.._gram.c.obj 
[10:07:57.853] "cl" "-Isrc\backend\parser\parser.a.p" "-Isrc\backend\parser" "-I..\src\backend\parser" "-Isrc\include" "-I..\src\include" "-Ic:\openssl\1.1\include" "-I..\src\include\port\win32" "-I..\src\include\port\win32_msvc" "/MDd" "/nologo" "/showIncludes" "/utf-8" "/W2" "/Od" "/Zi" "/DWIN32" "/DWINDOWS" "/D__WINDOWS__" "/D__WIN32__" "/D_CRT_SECURE_NO_DEPRECATE" "/D_CRT_NONSTDC_NO_DEPRECATE" "/wd4018" "/wd4244" "/wd4273" "/wd4101" "/wd4102" "/wd4090" "/wd4267" "-DBUILDING_DLL" "/Fdsrc\backend\parser\parser.a.p\meson-generated_.._gram.c.pdb" /Fosrc/backend/parser/parser.a.p/meson-generated_.._gram.c.obj "/c" src/backend/parser/gram.c
[10:07:57.860] c:\cirrus\build\src\backend\parser\gram.h(379): error C2365: 'PATTERN': redefinition; previous definition was 'typedef'
[10:07:57.860] C:\Program Files (x86)\Windows Kits\10\include\10.0.20348.0\um\wingdi.h(1375): note: see declaration of 'PATTERN'
[10:07:57.860] c:\cirrus\build\src\backend\parser\gram.h(379): error C2086: 'yytokentype PATTERN': redefinition
[10:07:57.860] c:\cirrus\build\src\backend\parser\gram.h(379): note: see declaration of 'PATTERN'
[10:07:57.860] ninja: build stopped: subcommand failed.

* Resolve complaint from "make headerscheck"

- Change Windowapi.h and nodeWindowAgg.c to remove unecessary extern
  and public functions.
  
Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  34. Re: Row pattern recognition

    Erik Rijkers <er@xs4all.nl> — 2023-09-02T18:04:02Z

    Op 9/2/23 om 08:52 schreef Tatsuo Ishii:

> Attached is the v5 patch. Differences from previous patch include:
> 

Hi,

The patches compile & tests run fine but this statement from the 
documentation crashes an assert-enabled server:

SELECT company, tdate, price, max(price) OVER w FROM stock
WINDOW w AS (
PARTITION BY company
ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
AFTER MATCH SKIP PAST LAST ROW
INITIAL
PATTERN (LOWPRICE UP+ DOWN+)
DEFINE
LOWPRICE AS price <= 100,
UP AS price > PREV(price),
DOWN AS price < PREV(price)
);
server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
connection to server was lost


Log file:

TRAP: failed Assert("aggregatedupto_nonrestarted <= 
winstate->aggregatedupto"), File: "nodeWindowAgg.c", Line: 1054, PID: 68975
postgres: 17_rpr_d0ec_gulo: aardvark testdb ::1(34808) 
SELECT(ExceptionalCondition+0x54)[0x9b0824]
postgres: 17_rpr_d0ec_gulo: aardvark testdb ::1(34808) SELECT[0x71ae8d]
postgres: 17_rpr_d0ec_gulo: aardvark testdb ::1(34808) 
SELECT(standard_ExecutorRun+0x13a)[0x6def9a]
/home/aardvark/pg_stuff/pg_installations/pgsql.rpr/lib/pg_stat_statements.so(+0x55e5)[0x7ff3798b95e5]
/home/aardvark/pg_stuff/pg_installations/pgsql.rpr/lib/auto_explain.so(+0x2680)[0x7ff3798ab680]
postgres: 17_rpr_d0ec_gulo: aardvark testdb ::1(34808) SELECT[0x88a4ff]
postgres: 17_rpr_d0ec_gulo: aardvark testdb ::1(34808) 
SELECT(PortalRun+0x240)[0x88bb50]
postgres: 17_rpr_d0ec_gulo: aardvark testdb ::1(34808) SELECT[0x887cca]
postgres: 17_rpr_d0ec_gulo: aardvark testdb ::1(34808) 
SELECT(PostgresMain+0x14dc)[0x88958c]
postgres: 17_rpr_d0ec_gulo: aardvark testdb ::1(34808) SELECT[0x7fb0da]
postgres: 17_rpr_d0ec_gulo: aardvark testdb ::1(34808) 
SELECT(PostmasterMain+0xd2d)[0x7fc01d]
postgres: 17_rpr_d0ec_gulo: aardvark testdb ::1(34808) 
SELECT(main+0x1e0)[0x5286d0]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7ff378e9dd0a]
postgres: 17_rpr_d0ec_gulo: aardvark testdb ::1(34808) 
SELECT(_start+0x2a)[0x5289aa]
2023-09-02 19:59:05.329 CEST 46723 LOG:  server process (PID 68975) was 
terminated by signal 6: Aborted
2023-09-02 19:59:05.329 CEST 46723 DETAIL:  Failed process was running: 
SELECT company, tdate, price, max(price) OVER w FROM stock
         WINDOW w AS (
         PARTITION BY company
         ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
         AFTER MATCH SKIP PAST LAST ROW
         INITIAL
         PATTERN (LOWPRICE UP+ DOWN+)
         DEFINE
         LOWPRICE AS price <= 100,
         UP AS price > PREV(price),
         DOWN AS price < PREV(price)
         );
2023-09-02 19:59:05.329 CEST 46723 LOG:  terminating any other active 
server processes



Erik Rijkers

  35. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-09-03T00:03:44Z

    > Hi,
> 
> The patches compile & tests run fine but this statement from the
> documentation crashes an assert-enabled server:
> 
> SELECT company, tdate, price, max(price) OVER w FROM stock
> WINDOW w AS (
> PARTITION BY company
> ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
> AFTER MATCH SKIP PAST LAST ROW
> INITIAL
> PATTERN (LOWPRICE UP+ DOWN+)
> DEFINE
> LOWPRICE AS price <= 100,
> UP AS price > PREV(price),
> DOWN AS price < PREV(price)
> );
> server closed the connection unexpectedly
> 	This probably means the server terminated abnormally
> 	before or while processing the request.
> connection to server was lost

Thank you for the report. Currently the patch has an issue with
aggregate functions including max. I have been working on aggregations
in row pattern recognition but will take more time to complete the
part.

In the mean time if you want to play with RPR, you can try window
functions. Examples can be found in src/test/regress/sql/rpr.sql.
Here is one of this:

-- the first row start with less than or equal to 100
SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w
 FROM stock
 WINDOW w AS (
 PARTITION BY company
 ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
 INITIAL
 PATTERN (LOWPRICE UP+ DOWN+)
 DEFINE
  LOWPRICE AS price <= 100,
  UP AS price > PREV(price),
  DOWN AS price < PREV(price)
);

-- second row raises 120%
SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w
 FROM stock
 WINDOW w AS (
 PARTITION BY company
 ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
 INITIAL
 PATTERN (LOWPRICE UP+ DOWN+)
 DEFINE
  LOWPRICE AS price <= 100,
  UP AS price > PREV(price) * 1.2,
  DOWN AS price < PREV(price)
);

Sorry for inconvenience.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  36. Re: Row pattern recognition

    Jacob Champion <jchampion@timescale.com> — 2023-09-08T00:00:07Z

    Hello!

> (1) I completely changed the pattern matching engine so that it
> performs backtracking. Now the engine evaluates all pattern elements
> defined in PATTER against each row, saving matched pattern variables
> in a string per row. For example if the pattern element A and B
> evaluated to true, a string "AB" is created for current row.

If I understand correctly, this strategy assumes that one row's
membership in a pattern variable is independent of the other rows'
membership. But I don't think that's necessarily true:

    DEFINE
      A AS PREV(CLASSIFIER()) IS DISTINCT FROM 'A',
      ...

> See row_is_in_reduced_frame, search_str_set and search_str_set_recurse
> in nodeWindowAggs.c for more details. For now I use a naive depth
> first search and probably there is a lot of rooms for optimization
> (for example rewriting it without using
> recursion).

The depth-first match is doing a lot of subtle work here. For example, with

    PATTERN ( A+ B+ )
    DEFINE A AS TRUE,
           B AS TRUE

(i.e. all rows match both variables), and three rows in the partition,
our candidates will be tried in the order

    aaa
    aab
    aba
    abb
    ...
    bbb

The only possible matches against our regex `^a+b+` are "aab" and "abb",
and that order matches the preferment order, so it's fine. But it's easy
to come up with a pattern where that's the wrong order, like

    PATTERN ( A+ (B|A)+ )

Now "aaa" will be considered before "aab", which isn't correct.

Similarly, the assumption that we want to match the longest string only
works because we don't allow alternation yet.

> Suggestions/patches are welcome.

Cool, I will give this piece some more thought. Do you mind if I try to
add some more complicated pattern quantifiers to stress the
architecture, or would you prefer to tackle that later? Just alternation
by itself will open up a world of corner cases.

> With the new engine, cases above do not fail anymore. See new
> regression test cases. Thanks for providing valuable test cases!

You're very welcome!

On 8/9/23 01:41, Tatsuo Ishii wrote:
> - I found a bug with pattern matching code. It creates a string for
>   subsequent regular expression matching. It uses the initial letter
>   of each define variable name. For example, if the varname is "foo",
>   then "f" is used. Obviously this makes trouble if we have two or
>   more variables starting with same "f" (e.g. "food"). To fix this, I
>   assign [a-z] to each variable instead of its initial letter. However
>   this way limits us not to have more than 26 variables. I hope 26 is
>   enough for most use cases.

There are still plenty of alphanumerics left that could be assigned...

But I'm wondering if we might want to just implement the NFA directly?
The current implementation's Cartesian explosion can probably be pruned
aggressively, but replaying the entire regex match once for every
backtracked step will still duplicate a lot of work.

--

I've attached another test case; it looks like last_value() is depending
on some sort of side effect from either first_value() or nth_value(). I
know the window frame itself is still under construction, so apologies
if this is an expected failure.

Thanks!
--Jacob

  37. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-09-08T03:54:47Z

    Hi,

> Hello!
> 
>> (1) I completely changed the pattern matching engine so that it
>> performs backtracking. Now the engine evaluates all pattern elements
>> defined in PATTER against each row, saving matched pattern variables
>> in a string per row. For example if the pattern element A and B
>> evaluated to true, a string "AB" is created for current row.
> 
> If I understand correctly, this strategy assumes that one row's
> membership in a pattern variable is independent of the other rows'
> membership. But I don't think that's necessarily true:
> 
>     DEFINE
>       A AS PREV(CLASSIFIER()) IS DISTINCT FROM 'A',
>       ...

But:

UP AS price > PREV(price)

also depends on previous row, no? Can you please elaborate how your
example could break current implementation? I cannot test it because
CLASSIFIER is not implemented yet.

>> See row_is_in_reduced_frame, search_str_set and search_str_set_recurse
>> in nodeWindowAggs.c for more details. For now I use a naive depth
>> first search and probably there is a lot of rooms for optimization
>> (for example rewriting it without using
>> recursion).
> 
> The depth-first match is doing a lot of subtle work here. For example, with
> 
>     PATTERN ( A+ B+ )
>     DEFINE A AS TRUE,
>            B AS TRUE
> 
> (i.e. all rows match both variables), and three rows in the partition,
> our candidates will be tried in the order
> 
>     aaa
>     aab
>     aba
>     abb
>     ...
>     bbb
> 
> The only possible matches against our regex `^a+b+` are "aab" and "abb",
> and that order matches the preferment order, so it's fine. But it's easy
> to come up with a pattern where that's the wrong order, like
> 
>     PATTERN ( A+ (B|A)+ )
> 
> Now "aaa" will be considered before "aab", which isn't correct.

Can you explain a little bit more? I think 'aaa' matches a regular
expression 'a+(b|a)+' and should be no problem before "aab" is
considered.

> Similarly, the assumption that we want to match the longest string only
> works because we don't allow alternation yet.

Can you please clarify more on this?

> Cool, I will give this piece some more thought. Do you mind if I try to
> add some more complicated pattern quantifiers to stress the
> architecture, or would you prefer to tackle that later? Just alternation
> by itself will open up a world of corner cases.

Do you mean you want to provide a better patch for the pattern
matching part? That will be helpfull. Because I am currently working
on the aggregation part and have no time to do it. However, the
aggregation work affects the v5 patch: it needs a refactoring. So can
you wait until I release v6 patch? I hope it will be released in two
weeks or so.

> On 8/9/23 01:41, Tatsuo Ishii wrote:
>> - I found a bug with pattern matching code. It creates a string for
>>   subsequent regular expression matching. It uses the initial letter
>>   of each define variable name. For example, if the varname is "foo",
>>   then "f" is used. Obviously this makes trouble if we have two or
>>   more variables starting with same "f" (e.g. "food"). To fix this, I
>>   assign [a-z] to each variable instead of its initial letter. However
>>   this way limits us not to have more than 26 variables. I hope 26 is
>>   enough for most use cases.
> 
> There are still plenty of alphanumerics left that could be assigned...
> 
> But I'm wondering if we might want to just implement the NFA directly?
> The current implementation's Cartesian explosion can probably be pruned
> aggressively, but replaying the entire regex match once for every
> backtracked step will still duplicate a lot of work.

Not sure if you mean implementing new regular expression engine
besides src/backend/regexp. I am afraid it's not a trivial work. The
current regexp code consists of over 10k lines. What do you think?

> I've attached another test case; it looks like last_value() is depending
> on some sort of side effect from either first_value() or nth_value(). I
> know the window frame itself is still under construction, so apologies
> if this is an expected failure.

Thanks. Fortunately current code which I am working passes the new
test. I will include it in the next (v6) patch.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  38. Re: Row pattern recognition

    Jacob Champion <jchampion@timescale.com> — 2023-09-08T19:27:05Z

    On 9/7/23 20:54, Tatsuo Ishii wrote:
>>     DEFINE
>>       A AS PREV(CLASSIFIER()) IS DISTINCT FROM 'A',
>>       ...
> 
> But:
> 
> UP AS price > PREV(price)
> 
> also depends on previous row, no?

PREV(CLASSIFIER()) depends not on the value of the previous row but the
state of the match so far. To take an example from the patch:

> * Example:
> * str_set[0] = "AB";
> * str_set[1] = "AC";
> * In this case at row 0 A and B are true, and A and C are true in row 1.

With these str_sets and my example DEFINE, row[1] is only classifiable
as 'A' if row[0] is *not* classified as 'A' at this point in the match.
"AA" is not a valid candidate string, even if it matches the PATTERN.

So if we don't reevaluate the pattern variable condition for the row, we
at least have to prune the combinations that search_str_set() visits, so
that we don't generate a logically impossible combination. That seems
like it could be pretty fragile, and it may be difficult for us to prove
compliance.

>> But it's easy
>> to come up with a pattern where that's the wrong order, like
>>
>>     PATTERN ( A+ (B|A)+ )
>>
>> Now "aaa" will be considered before "aab", which isn't correct.
> 
> Can you explain a little bit more? I think 'aaa' matches a regular
> expression 'a+(b|a)+' and should be no problem before "aab" is
> considered.

Assuming I've understood the rules correctly, we're not allowed to
classify the last row as 'A' if it also matches 'B'. Lexicographic
ordering takes precedence, so we have to try "aab" first. Otherwise our
query could return different results compared to another implementation.

>> Similarly, the assumption that we want to match the longest string only
>> works because we don't allow alternation yet.
> 
> Can you please clarify more on this?

Sure: for the pattern

    PATTERN ( (A|B)+ )

we have to consider the candidate "a" over the candidate "ba", even
though the latter is longer. Like the prior example, lexicographic
ordering is considered more important than the greedy quantifier.
Quoting ISO/IEC 9075-2:2016:

    More precisely, with both reluctant and greedy quantifiers, the set
    of matches is ordered lexicographically, but when one match is an
    initial substring of another match, reluctant quantifiers prefer the
    shorter match (the substring), whereas greedy quantifiers prefer the
    longer match (the “superstring”).

Here, "ba" doesn't have "a" as a prefix, so "ba" doesn't get priority.
ISO/IEC 19075-5:2021 has a big section on this (7.2) with worked examples.

(The "lexicographic order matters more than greediness" concept was the
most mind-bending part for me so far, probably because I haven't figured
out how to translate the concept into POSIX EREs. It wouldn't make sense
to say "the letter 't' can match 'a', 'B', or '3' in this regex", but
that's what RPR is doing.)

>> Cool, I will give this piece some more thought. Do you mind if I try to
>> add some more complicated pattern quantifiers to stress the
>> architecture, or would you prefer to tackle that later? Just alternation
>> by itself will open up a world of corner cases.
> 
> Do you mean you want to provide a better patch for the pattern
> matching part? That will be helpfull.

No guarantees that I'll find a better patch :D But yes, I will give it a
try.

> Because I am currently working
> on the aggregation part and have no time to do it. However, the
> aggregation work affects the v5 patch: it needs a refactoring. So can
> you wait until I release v6 patch? I hope it will be released in two
> weeks or so.

Absolutely!

>> But I'm wondering if we might want to just implement the NFA directly?
>> The current implementation's Cartesian explosion can probably be pruned
>> aggressively, but replaying the entire regex match once for every
>> backtracked step will still duplicate a lot of work.
> 
> Not sure if you mean implementing new regular expression engine
> besides src/backend/regexp. I am afraid it's not a trivial work. The
> current regexp code consists of over 10k lines. What do you think?

Heh, I think it would be pretty foolish for me to code an NFA, from
scratch, and then try to convince the community to maintain it.

But:
- I think we have to implement a parallel parser regardless (RPR PATTERN
syntax looks incompatible with POSIX)
- I suspect we need more control over the backtracking than the current
pg_reg* API is going to give us, or else I'm worried performance is
going to fall off a cliff with usefully-large partitions
- there's a lot of stuff in POSIX EREs that we don't need, and of the
features we do need, the + quantifier is probably one of the easiest
- it seems easier to prove the correctness of a slow, naive,
row-at-a-time engine, because we can compare it directly to the spec

So what I'm thinking is: if I start by open-coding the + quantifier, and
slowly add more pieces in, then it might be easier to see the parts of
src/backend/regex that I've duplicated. We can try to expose those parts
directly from the internal API to replace my bad implementation. And if
there are parts that aren't duplicated, then it'll be easier to explain
why we need something different from the current engine.

Does that seem like a workable approach? (Worst-case, my code is just
horrible, and we throw it in the trash.)

>> I've attached another test case; it looks like last_value() is depending
>> on some sort of side effect from either first_value() or nth_value(). I
>> know the window frame itself is still under construction, so apologies
>> if this is an expected failure.
> 
> Thanks. Fortunately current code which I am working passes the new
> test. I will include it in the next (v6) patch.

Great!

Thanks,
--Jacob

  39. Re: Row pattern recognition

    Vik Fearing <vik@postgresfriends.org> — 2023-09-08T21:43:10Z

    On 9/8/23 21:27, Jacob Champion wrote:
> On 9/7/23 20:54, Tatsuo Ishii wrote:

>>> But it's easy
>>> to come up with a pattern where that's the wrong order, like
>>>
>>>      PATTERN ( A+ (B|A)+ )
>>>
>>> Now "aaa" will be considered before "aab", which isn't correct.
>>
>> Can you explain a little bit more? I think 'aaa' matches a regular
>> expression 'a+(b|a)+' and should be no problem before "aab" is
>> considered.
> 
> Assuming I've understood the rules correctly, we're not allowed to
> classify the last row as 'A' if it also matches 'B'. Lexicographic
> ordering takes precedence, so we have to try "aab" first. Otherwise our
> query could return different results compared to another implementation.


Your understanding is correct.
-- 
Vik Fearing

  40. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-09-09T11:21:21Z

    Hi,

>> But:
>> 
>> UP AS price > PREV(price)
>> 
>> also depends on previous row, no?
> 
> PREV(CLASSIFIER()) depends not on the value of the previous row but the
> state of the match so far. To take an example from the patch:
> 
>> * Example:
>> * str_set[0] = "AB";
>> * str_set[1] = "AC";
>> * In this case at row 0 A and B are true, and A and C are true in row 1.
> 
> With these str_sets and my example DEFINE, row[1] is only classifiable
> as 'A' if row[0] is *not* classified as 'A' at this point in the match.
> "AA" is not a valid candidate string, even if it matches the PATTERN.

Ok, Let me clarify my understanding. Suppose we have:

PATTER (A B)
DEFINE A AS PREV(CLASSIFIER()) IS DISTINCT FROM 'A',
B AS price > 100

and the target table has price column values:

row[0]: 110
row[1]: 110
row[2]: 110
row[3]: 110

Then we will get for str_set:
r0: B
r1: AB

Because r0 only has classifier B, r1 can have A and B.  Problem is,
r2. If we choose A at r1, then r2 = B. But if we choose B at t1, then
r2 = AB. I guess this is the issue you pointed out.

> So if we don't reevaluate the pattern variable condition for the row, we
> at least have to prune the combinations that search_str_set() visits, so
> that we don't generate a logically impossible combination. That seems
> like it could be pretty fragile, and it may be difficult for us to prove
> compliance.

Yeah, probably we have delay evaluation of such pattern variables like
A, then reevaluate A after the first scan.

What about leaving this (reevaluation) for now? Because:

1) we don't have CLASSIFIER
2) we don't allow to give CLASSIFIER to PREV as its arggument

so I think we don't need to worry about this for now.

>> Can you explain a little bit more? I think 'aaa' matches a regular
>> expression 'a+(b|a)+' and should be no problem before "aab" is
>> considered.
> 
> Assuming I've understood the rules correctly, we're not allowed to
> classify the last row as 'A' if it also matches 'B'. Lexicographic
> ordering takes precedence, so we have to try "aab" first. Otherwise our
> query could return different results compared to another implementation.
> 
>>> Similarly, the assumption that we want to match the longest string only
>>> works because we don't allow alternation yet.
>> 
>> Can you please clarify more on this?
> 
> Sure: for the pattern
> 
>     PATTERN ( (A|B)+ )
> 
> we have to consider the candidate "a" over the candidate "ba", even
> though the latter is longer. Like the prior example, lexicographic
> ordering is considered more important than the greedy quantifier.
> Quoting ISO/IEC 9075-2:2016:
> 
>     More precisely, with both reluctant and greedy quantifiers, the set
>     of matches is ordered lexicographically, but when one match is an
>     initial substring of another match, reluctant quantifiers prefer the
>     shorter match (the substring), whereas greedy quantifiers prefer the
>     longer match (the “superstring”).
> 
> Here, "ba" doesn't have "a" as a prefix, so "ba" doesn't get priority.
> ISO/IEC 19075-5:2021 has a big section on this (7.2) with worked examples.
> 
> (The "lexicographic order matters more than greediness" concept was the
> most mind-bending part for me so far, probably because I haven't figured
> out how to translate the concept into POSIX EREs. It wouldn't make sense
> to say "the letter 't' can match 'a', 'B', or '3' in this regex", but
> that's what RPR is doing.)

Thanks for the explanation.  Surprising concet of the standard:-)　Is
it different from SIMILAR TO REs too?

What if we don't follow the standard, instead we follow POSIX EREs?  I
think this is better for users unless RPR's REs has significant merit
for users.

>> Do you mean you want to provide a better patch for the pattern
>> matching part? That will be helpfull.
> 
> No guarantees that I'll find a better patch :D But yes, I will give it a
> try.

Ok.

>> Because I am currently working
>> on the aggregation part and have no time to do it. However, the
>> aggregation work affects the v5 patch: it needs a refactoring. So can
>> you wait until I release v6 patch? I hope it will be released in two
>> weeks or so.
> 
> Absolutely!

Thanks.

> Heh, I think it would be pretty foolish for me to code an NFA, from
> scratch, and then try to convince the community to maintain it.
> 
> But:
> - I think we have to implement a parallel parser regardless (RPR PATTERN
> syntax looks incompatible with POSIX)

I am not sure if we need to worry about this because of the reason I
mentioned above.

> - I suspect we need more control over the backtracking than the current
> pg_reg* API is going to give us, or else I'm worried performance is
> going to fall off a cliff with usefully-large partitions

Agreed.

> - there's a lot of stuff in POSIX EREs that we don't need, and of the
> features we do need, the + quantifier is probably one of the easiest
> - it seems easier to prove the correctness of a slow, naive,
> row-at-a-time engine, because we can compare it directly to the spec
> 
> So what I'm thinking is: if I start by open-coding the + quantifier, and
> slowly add more pieces in, then it might be easier to see the parts of
> src/backend/regex that I've duplicated. We can try to expose those parts
> directly from the internal API to replace my bad implementation. And if
> there are parts that aren't duplicated, then it'll be easier to explain
> why we need something different from the current engine.
> 
> Does that seem like a workable approach? (Worst-case, my code is just
> horrible, and we throw it in the trash.)

Yes, it seems workable. I think for the first cut of RPR needs at
least the +quantifier with reasonable performance. The current naive
implementation seems to have issue because of exhaustive search.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  41. Re: Row pattern recognition

    Vik Fearing <vik@postgresfriends.org> — 2023-09-09T13:32:41Z

    On 9/9/23 13:21, Tatsuo Ishii wrote:
> Thanks for the explanation.  Surprising concet of the standard:-)

<quote from 19075-5:2023>

This leaves the choice between traditional NFA and Posix NFA. The 
difference between these is that a traditional NFA exits (declares a 
match) as soon as it finds the first possible match, whereas a Posix NFA 
is obliged to find all possible matches and then choose the “leftmost 
longest”. There are examples that show that, even for conventional 
regular expression matching on text strings and without back references, 
there are patterns for which a Posix NFA is orders of magnitude slower 
than a traditional NFA. In addition, reluctant quantifiers cannot be 
defined in a Posix NFA, because of the leftmost longest rule.

Therefore it was decided not to use the Posix NFA model, which leaves 
the traditional NFA as the model for row pattern matching. Among 
available tools that use traditional NFA engines, Perl is the most 
influential; therefore Perl was adopted as the design target for pattern 
matching rules.

</quote>

> Is it different from SIMILAR TO REs too?

Of course it is. :-)  SIMILAR TO uses its own language and rules.

> What if we don't follow the standard, instead we follow POSIX EREs?  I
> think this is better for users unless RPR's REs has significant merit
> for users.

This would get big pushback from me.
-- 
Vik Fearing

  42. Re: Row pattern recognition

    Jacob Champion <jchampion@timescale.com> — 2023-09-11T22:13:43Z

    On Sat, Sep 9, 2023 at 4:21 AM Tatsuo Ishii <ishii@sraoss.co.jp> wrote:
> Then we will get for str_set:
> r0: B
> r1: AB
>
> Because r0 only has classifier B, r1 can have A and B.  Problem is,
> r2. If we choose A at r1, then r2 = B. But if we choose B at t1, then
> r2 = AB. I guess this is the issue you pointed out.

Right.

> Yeah, probably we have delay evaluation of such pattern variables like
> A, then reevaluate A after the first scan.
>
> What about leaving this (reevaluation) for now? Because:
>
> 1) we don't have CLASSIFIER
> 2) we don't allow to give CLASSIFIER to PREV as its arggument
>
> so I think we don't need to worry about this for now.

Sure. I'm all for deferring features to make it easier to iterate; I
just want to make sure the architecture doesn't hit a dead end. Or at
least, not without being aware of it.

Also: is CLASSIFIER the only way to run into this issue?

> What if we don't follow the standard, instead we follow POSIX EREs?  I
> think this is better for users unless RPR's REs has significant merit
> for users.

Piggybacking off of what Vik wrote upthread, I think we would not be
doing ourselves any favors by introducing a non-compliant
implementation that performs worse than a traditional NFA. Those would
be some awful bug reports.

> > - I think we have to implement a parallel parser regardless (RPR PATTERN
> > syntax looks incompatible with POSIX)
>
> I am not sure if we need to worry about this because of the reason I
> mentioned above.

Even if we adopted POSIX NFA semantics, we'd still have to implement
our own parser for the PATTERN part of the query. I don't think
there's a good way for us to reuse the parser in src/backend/regex.

> > Does that seem like a workable approach? (Worst-case, my code is just
> > horrible, and we throw it in the trash.)
>
> Yes, it seems workable. I think for the first cut of RPR needs at
> least the +quantifier with reasonable performance. The current naive
> implementation seems to have issue because of exhaustive search.

+1

Thanks!
--Jacob

  43. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-09-12T06:18:43Z

    >> What about leaving this (reevaluation) for now? Because:
>>
>> 1) we don't have CLASSIFIER
>> 2) we don't allow to give CLASSIFIER to PREV as its arggument
>>
>> so I think we don't need to worry about this for now.
> 
> Sure. I'm all for deferring features to make it easier to iterate; I
> just want to make sure the architecture doesn't hit a dead end. Or at
> least, not without being aware of it.

Ok, let's defer this issue. Currently the patch already exceeds 3k
lines. I am afraid too big patch cannot be reviewed by anyone, which
means it will never be committed.

> Also: is CLASSIFIER the only way to run into this issue?

Good question. I would like to know.

>> What if we don't follow the standard, instead we follow POSIX EREs?  I
>> think this is better for users unless RPR's REs has significant merit
>> for users.
> 
> Piggybacking off of what Vik wrote upthread, I think we would not be
> doing ourselves any favors by introducing a non-compliant
> implementation that performs worse than a traditional NFA. Those would
> be some awful bug reports.

What I am not sure about is, you and Vik mentioned that the
traditional NFA is superior that POSIX NFA in terms of performance.
But how "lexicographic ordering" is related to performance?

>> I am not sure if we need to worry about this because of the reason I
>> mentioned above.
> 
> Even if we adopted POSIX NFA semantics, we'd still have to implement
> our own parser for the PATTERN part of the query. I don't think
> there's a good way for us to reuse the parser in src/backend/regex.

Ok.

>> > Does that seem like a workable approach? (Worst-case, my code is just
>> > horrible, and we throw it in the trash.)
>>
>> Yes, it seems workable. I think for the first cut of RPR needs at
>> least the +quantifier with reasonable performance. The current naive
>> implementation seems to have issue because of exhaustive search.
> 
> +1

BTW, attched is the v6 patch. The differences from v5 include:

- Now aggregates can be used with RPR. Below is an example from the
  regression test cases, which is added by v6 patch.

- Fix assersion error pointed out by Erik.

SELECT company, tdate, price,
 first_value(price) OVER w,
 last_value(price) OVER w,
 max(price) OVER w,
 min(price) OVER w,
 sum(price) OVER w,
 avg(price) OVER w,
 count(price) OVER w
FROM stock
WINDOW w AS (
PARTITION BY company
ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
AFTER MATCH SKIP PAST LAST ROW
INITIAL
PATTERN (START UP+ DOWN+)
DEFINE
START AS TRUE,
UP AS price > PREV(price),
DOWN AS price < PREV(price)
);
 company  |   tdate    | price | first_value | last_value | max  | min | sum  |          avg          | count 
----------+------------+-------+-------------+------------+------+-----+------+-----------------------+-------
 company1 | 07-01-2023 |   100 |         100 |        140 |  200 | 100 |  590 |  147.5000000000000000 |     4
 company1 | 07-02-2023 |   200 |             |            |      |     |      |                       |      
 company1 | 07-03-2023 |   150 |             |            |      |     |      |                       |      
 company1 | 07-04-2023 |   140 |             |            |      |     |      |                       |      
 company1 | 07-05-2023 |   150 |             |            |      |     |      |                       |      
 company1 | 07-06-2023 |    90 |          90 |        120 |  130 |  90 |  450 |  112.5000000000000000 |     4
 company1 | 07-07-2023 |   110 |             |            |      |     |      |                       |      
 company1 | 07-08-2023 |   130 |             |            |      |     |      |                       |      
 company1 | 07-09-2023 |   120 |             |            |      |     |      |                       |      
 company1 | 07-10-2023 |   130 |             |            |      |     |      |                       |      
 company2 | 07-01-2023 |    50 |          50 |       1400 | 2000 |  50 | 4950 | 1237.5000000000000000 |     4
 company2 | 07-02-2023 |  2000 |             |            |      |     |      |                       |      
 company2 | 07-03-2023 |  1500 |             |            |      |     |      |                       |      
 company2 | 07-04-2023 |  1400 |             |            |      |     |      |                       |      
 company2 | 07-05-2023 |  1500 |             |            |      |     |      |                       |      
 company2 | 07-06-2023 |    60 |          60 |       1200 | 1300 |  60 | 3660 |  915.0000000000000000 |     4
 company2 | 07-07-2023 |  1100 |             |            |      |     |      |                       |      
 company2 | 07-08-2023 |  1300 |             |            |      |     |      |                       |      
 company2 | 07-09-2023 |  1200 |             |            |      |     |      |                       |      
 company2 | 07-10-2023 |  1300 |             |            |      |     |      |                       |      
(20 rows)

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  44. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-09-12T08:44:57Z

    Regarding v6 patch:

> SELECT company, tdate, price,
>  first_value(price) OVER w,
>  last_value(price) OVER w,
>  max(price) OVER w,
>  min(price) OVER w,
>  sum(price) OVER w,
>  avg(price) OVER w,
>  count(price) OVER w
> FROM stock
> WINDOW w AS (
> PARTITION BY company
> ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
> AFTER MATCH SKIP PAST LAST ROW
> INITIAL
> PATTERN (START UP+ DOWN+)
> DEFINE
> START AS TRUE,
> UP AS price > PREV(price),
> DOWN AS price < PREV(price)
> );
>  company  |   tdate    | price | first_value | last_value | max  | min | sum  |          avg          | count 
> ----------+------------+-------+-------------+------------+------+-----+------+-----------------------+-------
>  company1 | 07-01-2023 |   100 |         100 |        140 |  200 | 100 |  590 |  147.5000000000000000 |     4
>  company1 | 07-02-2023 |   200 |             |            |      |     |      |                       |      
>  company1 | 07-03-2023 |   150 |             |            |      |     |      |                       |      
>  company1 | 07-04-2023 |   140 |             |            |      |     |      |                       |      
>  company1 | 07-05-2023 |   150 |             |            |      |     |      |                       |      
>  company1 | 07-06-2023 |    90 |          90 |        120 |  130 |  90 |  450 |  112.5000000000000000 |     4
>  company1 | 07-07-2023 |   110 |             |            |      |     |      |                       |      
>  company1 | 07-08-2023 |   130 |             |            |      |     |      |                       |      
>  company1 | 07-09-2023 |   120 |             |            |      |     |      |                       |      
>  company1 | 07-10-2023 |   130 |             |            |      |     |      |                       |      
>  company2 | 07-01-2023 |    50 |          50 |       1400 | 2000 |  50 | 4950 | 1237.5000000000000000 |     4
>  company2 | 07-02-2023 |  2000 |             |            |      |     |      |                       |      
>  company2 | 07-03-2023 |  1500 |             |            |      |     |      |                       |      
>  company2 | 07-04-2023 |  1400 |             |            |      |     |      |                       |      
>  company2 | 07-05-2023 |  1500 |             |            |      |     |      |                       |      
>  company2 | 07-06-2023 |    60 |          60 |       1200 | 1300 |  60 | 3660 |  915.0000000000000000 |     4
>  company2 | 07-07-2023 |  1100 |             |            |      |     |      |                       |      
>  company2 | 07-08-2023 |  1300 |             |            |      |     |      |                       |      
>  company2 | 07-09-2023 |  1200 |             |            |      |     |      |                       |      
>  company2 | 07-10-2023 |  1300 |             |            |      |     |      |                       |      
> (20 rows)

count column for unmatched rows should have been 0, rather than
NULL. i.e.

 company  |   tdate    | price | first_value | last_value | max  | min | sum  |          avg          | count 
----------+------------+-------+-------------+------------+------+-----+------+-----------------------+-------
 company1 | 07-01-2023 |   100 |         100 |        140 |  200 | 100 |  590 |  147.5000000000000000 |     4
 company1 | 07-02-2023 |   200 |             |            |      |     |      |                       |      
 company1 | 07-03-2023 |   150 |             |            |      |     |      |                       |      
 company1 | 07-04-2023 |   140 |             |            |      |     |      |                       |      
 company1 | 07-05-2023 |   150 |             |            |      |     |      |                       |     0
 company1 | 07-06-2023 |    90 |          90 |        120 |  130 |  90 |  450 |  112.5000000000000000 |     4
 company1 | 07-07-2023 |   110 |             |            |      |     |      |                       |      
 company1 | 07-08-2023 |   130 |             |            |      |     |      |                       |      
 company1 | 07-09-2023 |   120 |             |            |      |     |      |                       |      
 company1 | 07-10-2023 |   130 |             |            |      |     |      |                       |     0
 company2 | 07-01-2023 |    50 |          50 |       1400 | 2000 |  50 | 4950 | 1237.5000000000000000 |     4
 company2 | 07-02-2023 |  2000 |             |            |      |     |      |                       |      
 company2 | 07-03-2023 |  1500 |             |            |      |     |      |                       |      
 company2 | 07-04-2023 |  1400 |             |            |      |     |      |                       |      
 company2 | 07-05-2023 |  1500 |             |            |      |     |      |                       |     0
 company2 | 07-06-2023 |    60 |          60 |       1200 | 1300 |  60 | 3660 |  915.0000000000000000 |     4
 company2 | 07-07-2023 |  1100 |             |            |      |     |      |                       |      
 company2 | 07-08-2023 |  1300 |             |            |      |     |      |                       |      
 company2 | 07-09-2023 |  1200 |             |            |      |     |      |                       |      
 company2 | 07-10-2023 |  1300 |             |            |      |     |      |                       |     0
(20 rows)

Attached is the fix against v6 patch. I will include this in upcoming v7 patch.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  45. Re: Row pattern recognition

    Jacob Champion <jchampion@timescale.com> — 2023-09-12T22:09:29Z

    On Mon, Sep 11, 2023 at 11:18 PM Tatsuo Ishii <ishii@sraoss.co.jp> wrote:
> What I am not sure about is, you and Vik mentioned that the
> traditional NFA is superior that POSIX NFA in terms of performance.
> But how "lexicographic ordering" is related to performance?

I think they're only tangentially related. POSIX NFAs have to fully
backtrack even after the first match is found, so that's where the
performance difference comes in. (We would be introducing new ways to
catastrophically backtrack if we used that approach.) But since you
don't visit every possible path through the graph with a traditional
NFA, it makes sense to define an order in which you visit the nodes,
so that you can reason about which string is actually going to be
matched in the end.

> BTW, attched is the v6 patch. The differences from v5 include:
>
> - Now aggregates can be used with RPR. Below is an example from the
>   regression test cases, which is added by v6 patch.

Great, thank you!

--Jacob

  46. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-09-13T05:14:07Z

    > <quote from 19075-5:2023>

I was looking for this but I only found ISO/IEC 19075-5:2021.
https://www.iso.org/standard/78936.html

Maybe 19075-5:2021 is the latest one?

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  47. Re: Row pattern recognition

    Vik Fearing <vik@postgresfriends.org> — 2023-09-13T11:28:45Z

    On 9/13/23 07:14, Tatsuo Ishii wrote:
>> <quote from 19075-5:2023>
> 
> I was looking for this but I only found ISO/IEC 19075-5:2021.
> https://www.iso.org/standard/78936.html
> 
> Maybe 19075-5:2021 is the latest one?

Yes, probably.  Sorry.
-- 
Vik Fearing

  48. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-09-13T12:35:53Z

    > On 9/13/23 07:14, Tatsuo Ishii wrote:
>>> <quote from 19075-5:2023>
>> I was looking for this but I only found ISO/IEC 19075-5:2021.
>> https://www.iso.org/standard/78936.html
>> Maybe 19075-5:2021 is the latest one?
> 
> Yes, probably.  Sorry.

No problem. Thanks for confirmation.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  49. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-09-22T05:16:40Z

    > Attached is the fix against v6 patch. I will include this in upcoming v7 patch.

Attached is the v7 patch. It includes the fix mentioned above.  Also
this time the pattern matching engine is enhanced: previously it
recursed to row direction, which means if the number of rows in a
frame is large, it could exceed the limit of stack depth.  The new
version recurses over matched pattern variables in a row, which is at
most 26 which should be small enough.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  50. Re: Row pattern recognition

    Erik Rijkers <er@xs4all.nl> — 2023-09-22T08:12:38Z

    Op 9/22/23 om 07:16 schreef Tatsuo Ishii:
>> Attached is the fix against v6 patch. I will include this in upcoming v7 patch.
> 
> Attached is the v7 patch. It includes the fix mentioned above.  Also

Hi,

In my hands, make check fails on the rpr test; see attached .diff file.
In these two statements:
-- using NEXT
-- using AFTER MATCH SKIP TO NEXT ROW
   result of first_value(price) and next_value(price) are empty.


Erik Rijkers


> this time the pattern matching engine is enhanced: previously it
> recursed to row direction, which means if the number of rows in a
> frame is large, it could exceed the limit of stack depth.  The new
> version recurses over matched pattern variables in a row, which is at
> most 26 which should be small enough.
> 
> Best reagards,
> --
> Tatsuo Ishii
> SRA OSS LLC
> English: http://www.sraoss.co.jp/index_en/
> Japanese:http://www.sraoss.co.jp

  51. Re: Row pattern recognition

    Erik Rijkers <er@xs4all.nl> — 2023-09-22T08:23:11Z

    Op 9/22/23 om 07:16 schreef Tatsuo Ishii:
>> Attached is the fix against v6 patch. I will include this in upcoming v7 patch.
> 
> Attached is the v7 patch. It includes the fix mentioned above.  Also
(Champion's address bounced; removed)

Hi,

In my hands, make check fails on the rpr test; see attached .diff file.
In these two statements:
-- using NEXT
-- using AFTER MATCH SKIP TO NEXT ROW
   result of first_value(price) and next_value(price) are empty.

Erik Rijkers


> this time the pattern matching engine is enhanced: previously it
> recursed to row direction, which means if the number of rows in a
> frame is large, it could exceed the limit of stack depth.  The new
> version recurses over matched pattern variables in a row, which is at
> most 26 which should be small enough.
> 
> Best reagards,
> --
> Tatsuo Ishii
> SRA OSS LLC
> English: http://www.sraoss.co.jp/index_en/
> Japanese:http://www.sraoss.co.jp

  52. Re: Row pattern recognition

    Erik Rijkers <er@xs4all.nl> — 2023-09-22T08:26:49Z

    Op 9/22/23 om 10:23 schreef Erik Rijkers:
> Op 9/22/23 om 07:16 schreef Tatsuo Ishii:
>>> Attached is the fix against v6 patch. I will include this in upcoming 
>>> v7 patch.
>>
>> Attached is the v7 patch. It includes the fix mentioned above.  Also
> (Champion's address bounced; removed)
> 

Sorry, I forgot to re-attach the regression.diffs with resend...

Erik

> Hi,
> 
> In my hands, make check fails on the rpr test; see attached .diff file.
> In these two statements:
> -- using NEXT
> -- using AFTER MATCH SKIP TO NEXT ROW
>    result of first_value(price) and next_value(price) are empty.
> 
> Erik Rijkers
> 
> 
>> this time the pattern matching engine is enhanced: previously it
>> recursed to row direction, which means if the number of rows in a
>> frame is large, it could exceed the limit of stack depth.  The new
>> version recurses over matched pattern variables in a row, which is at
>> most 26 which should be small enough.
>>
>> Best reagards,
>> -- 
>> Tatsuo Ishii
>> SRA OSS LLC
>> English: http://www.sraoss.co.jp/index_en/
>> Japanese:http://www.sraoss.co.jp
> 
>

  53. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-09-22T10:12:50Z

    > Op 9/22/23 om 07:16 schreef Tatsuo Ishii:
>>> Attached is the fix against v6 patch. I will include this in upcoming
>>> v7 patch.
>> Attached is the v7 patch. It includes the fix mentioned above.  Also
> (Champion's address bounced; removed)

On my side his adress bounced too:-<

> Hi,
> 
> In my hands, make check fails on the rpr test; see attached .diff
> file.
> In these two statements:
> -- using NEXT
> -- using AFTER MATCH SKIP TO NEXT ROW
>   result of first_value(price) and next_value(price) are empty.

Strange. I have checked out fresh master branch and applied the v7
patches, then ran make check. All tests including the rpr test
passed. This is Ubuntu 20.04.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  54. Re: Row pattern recognition

    Erik Rijkers <er@xs4all.nl> — 2023-09-22T11:28:11Z

    Op 9/22/23 om 12:12 schreef Tatsuo Ishii:
>> Op 9/22/23 om 07:16 schreef Tatsuo Ishii:
>>>> Attached is the fix against v6 patch. I will include this in upcoming
>>>> v7 patch.
>>> Attached is the v7 patch. It includes the fix mentioned above.  Also
>> (Champion's address bounced; removed)
> 
> On my side his adress bounced too:-<
> 
>> Hi,
>>
>> In my hands, make check fails on the rpr test; see attached .diff
>> file.
>> In these two statements:
>> -- using NEXT
>> -- using AFTER MATCH SKIP TO NEXT ROW
>>    result of first_value(price) and next_value(price) are empty.
> 
> Strange. I have checked out fresh master branch and applied the v7
> patches, then ran make check. All tests including the rpr test
> passed. This is Ubuntu 20.04.

The curious thing is that the server otherwise builds ok, and if I 
explicitly run on that server 'CREATE TEMP TABLE stock' + the 20 INSERTS 
  (just to make sure to have known data), those two statements now both 
return the correct result.

So maybe the testing/timing is wonky (not necessarily the server).

Erik

> 
> Best reagards,
> --
> Tatsuo Ishii
> SRA OSS LLC
> English: http://www.sraoss.co.jp/index_en/
> Japanese:http://www.sraoss.co.jp

  55. Re: Row pattern recognition

    Jacob Champion <champion.p@gmail.com> — 2023-09-22T14:48:22Z

    On Fri, Sep 22, 2023, 3:13 AM Tatsuo Ishii <ishii@sraoss.co.jp> wrote:

> > Op 9/22/23 om 07:16 schreef Tatsuo Ishii:
> >>> Attached is the fix against v6 patch. I will include this in upcoming
> >>> v7 patch.
> >> Attached is the v7 patch. It includes the fix mentioned above.  Also
> > (Champion's address bounced; removed)
>
> On my side his adress bounced too:-<
>

Yep. I'm still here, just lurking for now. It'll take a little time for me
to get back to this thread, as my schedule has changed significantly. :D

Thanks,
--Jacob

>

  56. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-09-25T05:26:30Z

    > On Fri, Sep 22, 2023, 3:13 AM Tatsuo Ishii <ishii@sraoss.co.jp> wrote:
> 
>> > Op 9/22/23 om 07:16 schreef Tatsuo Ishii:
>> >>> Attached is the fix against v6 patch. I will include this in upcoming
>> >>> v7 patch.
>> >> Attached is the v7 patch. It includes the fix mentioned above.  Also
>> > (Champion's address bounced; removed)
>>
>> On my side his adress bounced too:-<
>>
> 
> Yep. I'm still here, just lurking for now. It'll take a little time for me
> to get back to this thread, as my schedule has changed significantly. :D

Hope you get back soon...

By the way, I was thinking about eliminating recusrive calls in
pattern matching. Attached is the first cut of the implementation. In
the attached v8 patch:

- No recursive calls anymore. search_str_set_recurse was removed.

- Instead it generates all possible pattern variable name initial
  strings before pattern matching. Suppose we have "ab" (row 0) and
  "ac" (row 1). "a" and "b" represents the pattern variable names
  which are evaluated to true.  In this case it will generate "aa",
  "ac", "ba" and "bc" and they are examined by do_pattern_match one by
  one, which performs pattern matching.

- To implement this, an infrastructure string_set* are created. They
  take care of set of StringInfo.

I found that the previous implementation did not search all possible
cases. I believe the bug is fixed in v8.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  57. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-10-04T06:03:28Z

    > By the way, I was thinking about eliminating recusrive calls in
> pattern matching. Attached is the first cut of the implementation. In
> the attached v8 patch:
> 
> - No recursive calls anymore. search_str_set_recurse was removed.
> 
> - Instead it generates all possible pattern variable name initial
>   strings before pattern matching. Suppose we have "ab" (row 0) and
>   "ac" (row 1). "a" and "b" represents the pattern variable names
>   which are evaluated to true.  In this case it will generate "aa",
>   "ac", "ba" and "bc" and they are examined by do_pattern_match one by
>   one, which performs pattern matching.
> 
> - To implement this, an infrastructure string_set* are created. They
>   take care of set of StringInfo.
> 
> I found that the previous implementation did not search all possible
> cases. I believe the bug is fixed in v8.

The v8 patch does not apply anymore due to commit d060e921ea "Remove obsolete executor cleanup code".
So I rebased and created v9 patch. The differences from the v8 include:

- Fix bug with get_slots. It did not correctly detect the end of full frame.
- Add test case using "ROWS BETWEEN CURRENT ROW AND offset FOLLOWING".

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  58. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-10-22T02:39:20Z

    Attached is the v10 patch. This version enhances the performance of
pattern matching.  Previously it generated all possible pattern string
candidates. This resulted in unnecessarily large number of
candidates. For example if you have 2 pattern variables and the target
frame includes 100 rows, the number of candidates can reach to 2^100
in the worst case. To avoid this, I do a pruning in the v10
patch. Suppose you have:

PATTERN (A B+ C+)

Candidates like "BAC" "CAB" cannot survive because they never satisfy
the search pattern. To judge this, I assign sequence numbers (0, 1, 2)
to (A B C).  If the pattern generator tries to generate BA, this is
not allowed because the sequence number for B is 1 and for A is 0, and
0 < 1: B cannot be followed by A. Note that this technique can be
applied when the quantifiers are "+" or "*". Maybe other quantifiers
such as '?'  or '{n, m}' can be applied too but I don't confirm yet
because I have not implemented them yet.

Besides this improvement, I fixed a bug in the previous and older
patches: when an expression in DEFINE uses text operators, it errors
out:

ERROR:  could not determine which collation to use for string comparison
HINT:  Use the COLLATE clause to set the collation explicitly.

This was fixed by adding assign_expr_collations() in
transformDefineClause().

Also I have updated documentation "3.5. Window Functions"

- It still mentioned about rpr(). It's not applied anymore.
- Enhance the description about DEFINE and PATTERN.
- Mention that quantifier '*' is supported.

Finally I have added more test cases to the regression test.
- same pattern variable appears twice
- case for quantifier '*'

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  59. Re: Row pattern recognition

    Jacob Champion <champion.p@gmail.com> — 2023-10-24T18:51:19Z

    On Sat, Oct 21, 2023 at 7:39 PM Tatsuo Ishii <ishii@sraoss.co.jp> wrote:
> Attached is the v10 patch. This version enhances the performance of
> pattern matching.

Nice! I've attached a couple of more stressful tests (window
partitions of 1000 rows each). Beware that the second one runs my
desktop out of memory fairly quickly with the v10 implementation.

I was able to carve out some time this week to implement a very basic
recursive NFA, which handles both the + and * qualifiers (attached).
It's not production quality -- a frame on the call stack for every row
isn't going to work -- but with only those two features, it's pretty
tiny, and it's able to run the new stress tests with no issue. If I
continue to have time, I hope to keep updating this parallel
implementation as you add features to the StringSet implementation,
and we can see how it evolves. I expect that alternation and grouping
will ratchet up the complexity.

Thanks!
--Jacob

  60. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-10-25T00:11:05Z

    > On Sat, Oct 21, 2023 at 7:39 PM Tatsuo Ishii <ishii@sraoss.co.jp> wrote:
>> Attached is the v10 patch. This version enhances the performance of
>> pattern matching.
> 
> Nice! I've attached a couple of more stressful tests (window
> partitions of 1000 rows each). Beware that the second one runs my
> desktop out of memory fairly quickly with the v10 implementation.
> 
> I was able to carve out some time this week to implement a very basic
> recursive NFA, which handles both the + and * qualifiers (attached).

Great. I will look into this.

> It's not production quality -- a frame on the call stack for every row
> isn't going to work

Yeah.

> -- but with only those two features, it's pretty
> tiny, and it's able to run the new stress tests with no issue. If I
> continue to have time, I hope to keep updating this parallel
> implementation as you add features to the StringSet implementation,
> and we can see how it evolves. I expect that alternation and grouping
> will ratchet up the complexity.

Sounds like a plan.

By the way, I tested my patch (v10) to handle more large data set and
tried to following query with pgbench database. On my laptop it works
with 100k rows pgbench_accounts table but with beyond the number I got
OOM killer. I would like to enhance this in the next patch.

SELECT aid, first_value(aid) OVER w,
count(*) OVER w
FROM pgbench_accounts
WINDOW w AS (
PARTITION BY bid
ORDER BY aid
ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
AFTER MATCH SKIP PAST LAST ROW
INITIAL
PATTERN (START UP+)
DEFINE
START AS TRUE,
UP AS aid > PREV(aid)
);

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  61. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-10-25T02:49:30Z

    > Great. I will look into this.

I am impressed the simple NFA implementation.  It would be nicer if it
could be implemented without using recursion.

> By the way, I tested my patch (v10) to handle more large data set and
> tried to following query with pgbench database. On my laptop it works
> with 100k rows pgbench_accounts table but with beyond the number I got
       ~~~ I meant 10k.

> OOM killer. I would like to enhance this in the next patch.
> 
> SELECT aid, first_value(aid) OVER w,
> count(*) OVER w
> FROM pgbench_accounts
> WINDOW w AS (
> PARTITION BY bid
> ORDER BY aid
> ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
> AFTER MATCH SKIP PAST LAST ROW
> INITIAL
> PATTERN (START UP+)
> DEFINE
> START AS TRUE,
> UP AS aid > PREV(aid)
> );

I ran this against your patch. It failed around > 60k rows.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  62. Re: Row pattern recognition

    Jacob Champion <champion.p@gmail.com> — 2023-10-30T19:49:18Z

    On Tue, Oct 24, 2023 at 7:49 PM Tatsuo Ishii <ishii@sraoss.co.jp> wrote:
> I am impressed the simple NFA implementation.

Thanks!

> It would be nicer if it
> could be implemented without using recursion.

Yeah. If for some reason we end up going with a bespoke
implementation, I assume we'd just convert the algorithm to an
iterative one and optimize it heavily. But I didn't want to do that
too early, since it'd probably make it harder to add new features...
and anyway my goal is still to try to reuse src/backend/regex
eventually.

> > SELECT aid, first_value(aid) OVER w,
> > count(*) OVER w
> > FROM pgbench_accounts
> > WINDOW w AS (
> > PARTITION BY bid
> > ORDER BY aid
> > ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
> > AFTER MATCH SKIP PAST LAST ROW
> > INITIAL
> > PATTERN (START UP+)
> > DEFINE
> > START AS TRUE,
> > UP AS aid > PREV(aid)
> > );
>
> I ran this against your patch. It failed around > 60k rows.

Nice, that's actually more frames than I expected. Looks like I have
similar results here with my second test query (segfault at ~58k
rows).

Thanks,
--Jacob

  63. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-11-08T07:37:05Z

    >> It would be nicer if it
>> could be implemented without using recursion.
> 
> Yeah. If for some reason we end up going with a bespoke
> implementation, I assume we'd just convert the algorithm to an
> iterative one and optimize it heavily. But I didn't want to do that
> too early, since it'd probably make it harder to add new features...
> and anyway my goal is still to try to reuse src/backend/regex
> eventually.

Ok.

Attached is the v11 patch. Below are the summary of the changes from
previous version.

- rebase.

- Reduce memory allocation in pattern matching (search_str_set()). But
  still Champion's second stress test gives OOM killer.
    
  - While keeping an old set to next round, move the StringInfo to
    new_str_set, rather than copying from old_str_set. This allows to
    run pgbench.sql against up to 60k rows on my laptop (previously
    20k).
    
  - Use enlargeStringInfo to set the buffer size, rather than
    incrementally enlarge the buffer. This does not seem to give big
    enhancement but it should theoretically an enhancement.

- Fix "variable not found in subplan target list" error if WITH is
  used.
    
  - To fix this apply pullup_replace_vars() against DEFINE clause in
    planning phase (perform_pullup_replace_vars()).  Also add
    regression test cases for WITH that caused the error in the
    previous version.

- Fix the case when no greedy quantifiers ('+' or '*') are included in
  PATTERN.
    
  - Previously update_reduced_frame() did not consider the case and
    produced wrong results. Add another code path which is dedicated
    to none greedy PATTERN (at this point, it means there's no
    quantifier case). Also add a test case for this.

- Remove unnecessary check in transformPatternClause().

  - Previously it checked if all pattern variables are defined in
    DEFINE clause. But currently RPR allows to "auto define" such
    variables as "varname AS TRUE". So the check was not necessary.

- FYI here is the list to explain what was changed in each patch file.

0001-Row-pattern-recognition-patch-for-raw-parser.patch
- same

0002-Row-pattern-recognition-patch-parse-analysis.patch
- Add markTargetListOrigins() to transformFrameOffset().
- Change transformPatternClause().

0003-Row-pattern-recognition-patch-planner.patch
- Fix perform_pullup_replace_vars()

0004-Row-pattern-recognition-patch-executor.patch
- Fix update_reduced_frame()
- Fix search_str_set()

0005-Row-pattern-recognition-patch-docs.patch
- same

0006-Row-pattern-recognition-patch-tests.patch
- Add test case for non-greedy and WITH cases

0007-Allow-to-print-raw-parse-tree.patch
- same

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  64. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-12-08T01:16:13Z

    Sorry for posting v12 patch again. It seems the previous post of v12
patch email lost mail threading information and was not recognized as
a part of the thread by CF application and CFbot.
https://www.postgresql.org/message-id/20231204.204048.1998548830490453126.t-ishii%40sranhm.sra.co.jp

Attached is the v12 patch. Below are the summary of the changes from
previous version.

- Rebase. CFbot says v11 patch needs rebase since Nov 30, 2023.
 
- Apply preprocess_expression() to DEFINE clause in the planning
  phase.  This is necessary to simply const expressions like:

    DEFINE A price < (99 + 1)
    to:
    DEFINE A price < 100

- Re-allow to use WinSetMarkPosition() in eval_windowaggregates().

- FYI here is the list to explain what were changed in each patch file.

0001-Row-pattern-recognition-patch-for-raw-parser.patch
- Fix conflict.

0002-Row-pattern-recognition-patch-parse-analysis.patch
- Same as before.

0003-Row-pattern-recognition-patch-planner.patch
- Call preprocess_expression() for DEFINE clause in subquery_planner().

0004-Row-pattern-recognition-patch-executor.patch
- Re-allow to use WinSetMarkPosition() in eval_windowaggregates().

0005-Row-pattern-recognition-patch-docs.patch
- Same as before.

0006-Row-pattern-recognition-patch-tests.patch
- Same as before.

0007-Allow-to-print-raw-parse-tree.patch
- Same as before.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  65. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2023-12-08T22:22:58Z

    > On 04.12.23 12:40, Tatsuo Ishii wrote:
>> diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
>> index d631ac89a9..5a77fca17f 100644
>> --- a/src/backend/parser/gram.y
>> +++ b/src/backend/parser/gram.y
>> @@ -251,6 +251,8 @@ static Node *makeRecursiveViewSelect(char
>> *relname, List *aliases, Node *query);
>>   	DefElem	   *defelt;
>>   	SortBy	   *sortby;
>>   	WindowDef  *windef;
>> +	RPCommonSyntax	*rpcom;
>> +	RPSubsetItem	*rpsubset;
>>   	JoinExpr   *jexpr;
>>   	IndexElem  *ielem;
>>   	StatsElem  *selem;
>> @@ -278,6 +280,7 @@ static Node *makeRecursiveViewSelect(char
>> *relname, List *aliases, Node *query);
>>   	MergeWhenClause *mergewhen;
>>   	struct KeyActions *keyactions;
>>   	struct KeyAction *keyaction;
>> +	RPSkipTo	skipto;
>>   }
>>     %type <node>	stmt toplevel_stmt schema_stmt routine_body_stmt
> 
> It is usually not the style to add an entry for every node type to the
> %union.  Otherwise, we'd have hundreds of entries in there.

Ok, I have removed the node types and used existing node types.  Also
I have moved RPR related %types to same place to make it easier to know
what are added by RPR.

>> @@ -866,6 +878,7 @@ static Node *makeRecursiveViewSelect(char
>> *relname, List *aliases, Node *query);
>>   %nonassoc UNBOUNDED /* ideally would have same precedence as IDENT */
>>   %nonassoc IDENT PARTITION RANGE ROWS GROUPS PRECEDING FOLLOWING CUBE
>>   %ROLLUP
>>   			SET KEYS OBJECT_P SCALAR VALUE_P WITH WITHOUT
>> +%nonassoc	MEASURES AFTER INITIAL SEEK PATTERN_P
>>   %left Op OPERATOR /* multi-character ops and user-defined operators */
>>   %left		'+' '-'
>>   %left		'*' '/' '%'
> 
> It was recently discussed that these %nonassoc should ideally all have
> the same precedence.  Did you consider that here?

No, I didn't realize it. Thanks for pointing it out. I have removed
%nonassoc so that MEASURES etc. have the same precedence as IDENT etc.

Attached is the new diff of gram.y against master branch.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  66. Re: Row pattern recognition

    NINGWEI CHEN <chen@sraoss.co.jp> — 2024-01-22T05:51:49Z

    On Sat, 09 Dec 2023 07:22:58 +0900 (JST)
Tatsuo Ishii <ishii@sraoss.co.jp> wrote:

> > On 04.12.23 12:40, Tatsuo Ishii wrote:
> >> diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
> >> index d631ac89a9..5a77fca17f 100644
> >> --- a/src/backend/parser/gram.y
> >> +++ b/src/backend/parser/gram.y
> >> @@ -251,6 +251,8 @@ static Node *makeRecursiveViewSelect(char
> >> *relname, List *aliases, Node *query);
> >>   	DefElem	   *defelt;
> >>   	SortBy	   *sortby;
> >>   	WindowDef  *windef;
> >> +	RPCommonSyntax	*rpcom;
> >> +	RPSubsetItem	*rpsubset;
> >>   	JoinExpr   *jexpr;
> >>   	IndexElem  *ielem;
> >>   	StatsElem  *selem;
> >> @@ -278,6 +280,7 @@ static Node *makeRecursiveViewSelect(char
> >> *relname, List *aliases, Node *query);
> >>   	MergeWhenClause *mergewhen;
> >>   	struct KeyActions *keyactions;
> >>   	struct KeyAction *keyaction;
> >> +	RPSkipTo	skipto;
> >>   }
> >>     %type <node>	stmt toplevel_stmt schema_stmt routine_body_stmt
> > 
> > It is usually not the style to add an entry for every node type to the
> > %union.  Otherwise, we'd have hundreds of entries in there.
> 
> Ok, I have removed the node types and used existing node types.  Also
> I have moved RPR related %types to same place to make it easier to know
> what are added by RPR.
> 
> >> @@ -866,6 +878,7 @@ static Node *makeRecursiveViewSelect(char
> >> *relname, List *aliases, Node *query);
> >>   %nonassoc UNBOUNDED /* ideally would have same precedence as IDENT */
> >>   %nonassoc IDENT PARTITION RANGE ROWS GROUPS PRECEDING FOLLOWING CUBE
> >>   %ROLLUP
> >>   			SET KEYS OBJECT_P SCALAR VALUE_P WITH WITHOUT
> >> +%nonassoc	MEASURES AFTER INITIAL SEEK PATTERN_P
> >>   %left Op OPERATOR /* multi-character ops and user-defined operators */
> >>   %left		'+' '-'
> >>   %left		'*' '/' '%'
> > 
> > It was recently discussed that these %nonassoc should ideally all have
> > the same precedence.  Did you consider that here?
> 
> No, I didn't realize it. Thanks for pointing it out. I have removed
> %nonassoc so that MEASURES etc. have the same precedence as IDENT etc.
> 
> Attached is the new diff of gram.y against master branch.

Thank you very much for providing the patch for the RPR implementation.

After applying the v12-patches, I noticed an issue that
the rpr related parts in window clauses were not displayed in the
view definitions (the definition column of pg_views).

To address this, I have taken the liberty of adding an additional patch
that modifies the relevant rewriter source code.
I have attached the rewriter patch for your review and would greatly appreciate your feedback.

Thank you for your time and consideration.

-- 
SRA OSS LLC
Ningwei Chen <chen@sraoss.co.jp>
TEL: 03-5979-2701 FAX: 03-5979-2702

  67. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2024-01-22T06:22:11Z

    > Thank you very much for providing the patch for the RPR implementation.
> 
> After applying the v12-patches, I noticed an issue that
> the rpr related parts in window clauses were not displayed in the
> view definitions (the definition column of pg_views).
> 
> To address this, I have taken the liberty of adding an additional patch
> that modifies the relevant rewriter source code.
> I have attached the rewriter patch for your review and would greatly appreciate your feedback.
> 
> Thank you for your time and consideration.

Thank you so much for spotting the issue and creating the patch. I
confirmed that your patch applies cleanly and solve the issue. I will
include the patches into upcoming v13 patches.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  68. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2024-01-22T10:26:18Z

    Attached is the v13 patch. Below are the summary of the changes from
previous version (besides rebase).

0001-Row-pattern-recognition-patch-for-raw-parser.patch
- Fix raw paser per Peter Eisentraut's review. Remove the new node
  types and use existing ones. Also remove %nonassoc so that
  MEASURES etc. have the same precedence as IDENT etc.

Peter's comment:
> It is usually not the style to add an entry for every node type to the
> %union.  Otherwise, we'd have hundreds of entries in there.

> It was recently discussed that these %nonassoc should ideally all have
> the same precedence.  Did you consider that here?

0002-Row-pattern-recognition-patch-parse-analysis.patch
- Fix transformRPR so that SKIP variable name in the AFTER MATCH SKIP
  TO clause is tracked. This is added by Ningwei Chen.

0003-Row-pattern-recognition-patch-rewriter.patch
This is a new patch for rewriter. Contributed by Ningwei Chen.

Chen's comment:
> After applying the v12-patches, I noticed an issue that
> the rpr related parts in window clauses were not displayed in the
> view definitions (the definition column of pg_views).

0004-Row-pattern-recognition-patch-planner.patch
- same as before (previously it was 0003-Row-pattern-recognition-patch-planner.patch)

0005-Row-pattern-recognition-patch-executor.patch
- same as before (previously it was 0004-Row-pattern-recognition-patch-executor.patch)

0006-Row-pattern-recognition-patch-docs.patch
- Same as before. (previously it was 0005-Row-pattern-recognition-patch-docs.patch)

0007-Row-pattern-recognition-patch-tests.patch
- Same as before. (previously it was 0006-Row-pattern-recognition-patch-tests.patch)

0008-Allow-to-print-raw-parse-tree.patch
- Same as before. (previously it was 0007-Allow-to-print-raw-parse-tree.patch).
  Note that patch is not intended to be incorporated into main
  tree. This is just for debugging purpose. With this patch, raw parse
  tree is printed if debug_print_parse is enabled.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  69. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2024-02-29T00:19:54Z

    Attached is the v14 patch. Below are the summary of the changes from
previous version (besides rebase).
V14 patches are mainly for coding style fixes.

0001-Row-pattern-recognition-patch-for-raw-parser.patch
- Fold too long lines and run pgindent.

0002-Row-pattern-recognition-patch-parse-analysis.patch
- Fold too long lines and run pgindent.

0003-Row-pattern-recognition-patch-rewriter.patch
- Fold too long lines and run pgindent.

0004-Row-pattern-recognition-patch-planner.patch
- Fold too long lines and run pgindent.

0005-Row-pattern-recognition-patch-executor.patch
- Fold too long lines and run pgindent.

- Surround debug lines using "ifdef RPR_DEBUG" so that logs are not
  contaminated by RPR debug logs when log_min_messages are set to
  DEBUG1 or higher.

0006-Row-pattern-recognition-patch-docs.patch
- Same as before. (previously it was 0005-Row-pattern-recognition-patch-docs.patch)

0007-Row-pattern-recognition-patch-tests.patch
- Same as before. (previously it was 0006-Row-pattern-recognition-patch-tests.patch)

0008-Allow-to-print-raw-parse-tree.patch
- Same as before.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  70. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2024-03-28T10:59:25Z

    Attached is the v15 patch. No changes are made except rebasing due to
recent grammar changes.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  71. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2024-04-12T07:09:08Z

    Attached is the v16 patch. No changes are made except rebasing due to
recent grammar changes.

Also I removed chen@sraoss.co.jp from the Cc: list. The email address
is no longer valid.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  72. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2024-04-24T03:12:44Z

    Hi Vik and Champion,

I think the current RPR patch is not quite correct in handling
count(*).

(using slightly modified version of Vik's example query)

SELECT v.a, count(*) OVER w
FROM (VALUES ('A'),('B'),('B'),('C')) AS v (a)
WINDOW w AS (
  ORDER BY v.a
  ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
  PATTERN (B+)
  DEFINE B AS a = 'B'
)
 a | count 
---+-------
 A |     0
 B |     2
 B |      
 C |     0
(4 rows)

Here row 3 is skipped because the pattern B matches row 2 and 3. In
this case I think cont(*) should return 0 rathern than NULL for row 3.

What do you think?

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  73. Re: Row pattern recognition

    Jacob Champion <jacob.champion@enterprisedb.com> — 2024-04-24T17:55:29Z

    On Tue, Apr 23, 2024 at 8:13 PM Tatsuo Ishii <ishii@sraoss.co.jp> wrote:
> SELECT v.a, count(*) OVER w
> FROM (VALUES ('A'),('B'),('B'),('C')) AS v (a)
> WINDOW w AS (
>   ORDER BY v.a
>   ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>   PATTERN (B+)
>   DEFINE B AS a = 'B'
> )
>  a | count
> ---+-------
>  A |     0
>  B |     2
>  B |
>  C |     0
> (4 rows)
>
> Here row 3 is skipped because the pattern B matches row 2 and 3. In
> this case I think cont(*) should return 0 rathern than NULL for row 3.

I think returning zero would match Vik's explanation upthread [1],
yes. Unfortunately I don't have a spec handy to double-check for
myself right now.

--Jacob

[1] https://www.postgresql.org/message-id/c9ebc3d0-c3d1-e8eb-4a57-0ec099cbda17%40postgresfriends.org

  74. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2024-04-26T06:09:32Z

    > On Tue, Apr 23, 2024 at 8:13 PM Tatsuo Ishii <ishii@sraoss.co.jp> wrote:
>> SELECT v.a, count(*) OVER w
>> FROM (VALUES ('A'),('B'),('B'),('C')) AS v (a)
>> WINDOW w AS (
>>   ORDER BY v.a
>>   ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>>   PATTERN (B+)
>>   DEFINE B AS a = 'B'
>> )
>>  a | count
>> ---+-------
>>  A |     0
>>  B |     2
>>  B |
>>  C |     0
>> (4 rows)
>>
>> Here row 3 is skipped because the pattern B matches row 2 and 3. In
>> this case I think cont(*) should return 0 rathern than NULL for row 3.
> 
> I think returning zero would match Vik's explanation upthread [1],
> yes. Unfortunately I don't have a spec handy to double-check for
> myself right now.

Ok. I believe you and Vik are correct.
I am modifying the patch in this direction.
Attached is the regression diff after modifying the patch.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  75. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2024-04-28T11:28:26Z

    >> I think returning zero would match Vik's explanation upthread [1],
>> yes. Unfortunately I don't have a spec handy to double-check for
>> myself right now.
> 
> Ok. I believe you and Vik are correct.
> I am modifying the patch in this direction.

Attached are the v17 patches in the direction. Differences from v16
include:

- In 0005 executor patch, aggregation in RPR always restarts for each
  row. This is necessary to run aggregates on no matching (due to
  skipping) or empty matching (due to no pattern variables matches)
  rows to produce NULL (most aggregates) or 0 (count) properly. In v16
  I had a hack using a flag to force the aggregation results to be
  NULL in case of no match or empty match in
  finalize_windowaggregate(). v17 eliminates the dirty hack.

- 0006 docs and 0007 test patches are adjusted to reflect the RPR
  output chages in 0005.
  
Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  76. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2024-05-11T07:23:07Z

    Attached are the v18 patches. To fix conflicts due to recent commit:

7d2c7f08d9 Fix query pullup issue with WindowClause runCondition

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  77. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2024-05-15T00:02:03Z

    Attached are the v19 patches. Changes from v18 include:

0002:
- add a check whether DEFINE clause includes subqueries. If so, error out.
0007:
- fix wrong test (row pattern definition variable name must not appear
  more than once)
- remove unnessary test (undefined define variable is not allowed).
  We have already allowed the undefined variables.
- add tests: subqueries and aggregates in DEFINE clause are not
  supported. The standard allows them but I have not implemented them
  yet.

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  78. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2024-05-24T02:39:19Z

    Attached are the v20 patches. Just rebased.
(The conflict was in 0001 patch.)

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  79. Re: Row pattern recognition

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2024-06-13T00:25:01Z

    I gave a talk on RPR in PGConf.dev 2024.
https://www.pgevents.ca/events/pgconfdev2024/schedule/session/114-implementing-row-pattern-recognition/
(Slides are available from the link).

Vik Faring and Jacob Champion were one of the audiences and we had a
small discussion after the talk. We continued the discussion off list
on how to move forward the RPR implementation project. One of the
ideas is, to summarize what are in the patch and what are not from the
SQL standard specification's point of view. This should help us to
reach the consensus regarding "minimum viable" feature set if we want
to bring the patch in upcoming PostgreSQL v18.

Here is the first cut of the document. Comments/feedback are welcome.

-------------------------------------------------------------------------
This memo describes the current status of implementation of SQL/RPR
(Row Pattern Recognition), as of June 13, 2024 (the latest patch is v20).

- RPR in FROM clause and WINDOW clause

The SQL standard defines two features regarding SQL/RPR - R010 (RPR in
FROM clause) and R020 (RPR in WINDOW clause). Only R020 is
implemented. From now on, we discuss on R020.

- Overview of R020 syntax

WINDOW window_name AS (
[ PARTITION BY ... ]
[ ORDER BY... ]
[ MEASURES ... ]
ROWS BETWEEN CURRENT ROW AND ...
[ AFTER MATCH SKIP ... ]
[ INITIAL|SEEK ]
PATTERN (...)
[ SUBSET ... ]
DEFINE ...
)

-- PARTITION BY and ORDER BY are not specific to RPR and has been
  already there in current PostgreSQL.

-- What are (partially) implemented:

AFTER MATCH SKIP
INITIAL|SEEK
PATTERN
DEFINE

-- What are not implemented at all:
MEASURES
SUBSET

Followings are detailed status of the each clause.

- AFTER MATCH SKIP

-- Implemented:
AFTER MATCH SKIP TO NEXT ROW
AFTER MATCH SKIP PAST LAST ROW

-- Not implemented:
AFTER MATCH SKIP TO FIRST|LAST pattern_variable

- INITIAL|SEEK

--Implemented:
INITIAL

-- Not implemented:
SEEK

- DEFINE

-- Partially implemented row pattern navigation operations are PREV and
   NEXT. FIRST and LAST are not implemented.

-- The standard says PREV and NEXT accepts optional argument "offset"
   but it's not implemented.

-- The standard says the row pattern navigation operations can be
   nested but it's not implemented.

-- CLLASSIFIER, use of aggregate functions and subqueries in DEFINE
   clause are not implemented.

- PATTERN

-- Followings are implemented:
+: 1 or more rows
*: 0 or more rows

-- Followings are not implemented:
?: 0 or 1 row
A | B: OR condition
(A B): grouping
{n}: n rows
{n,}: n or more rows
{n,m}: greater or equal to n rows and less than or equal to m rows
{,m}: more than 0 and less than or equal to m rows
-------------------------------------------------------------------------

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  80. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2024-08-26T04:39:47Z

    Attached are the v21 patches. Just rebased.
(The conflict was in 0001 patch.)

The 0008 patch is just for debugging purpose. You can ignore it.
This hasn't been changed, but I would like to notice just in case.

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  81. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2024-09-19T04:59:47Z

    Attached are the v22 patches. Just rebased.  The conflict was in 0001
patch due to commit 89f908a6d0 "Add temporal FOREIGN KEY contraints".

The 0008 patch is just for debugging purpose. You can ignore it.
This hasn't been changed, but I would like to notice just in case.

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  82. Re: Row pattern recognition

    Jacob Champion <jacob.champion@enterprisedb.com> — 2024-09-27T22:27:07Z

    On Wed, Sep 18, 2024 at 10:00 PM Tatsuo Ishii <ishii@postgresql.org> wrote:
>
> Attached are the v22 patches. Just rebased.

Thanks!

With some bigger partitions, I hit an `ERROR:  wrong pos: 1024`. A
test that reproduces it is attached.

While playing with the feature, I've been trying to identify runs of
matched rows by eye. But it's pretty difficult -- the best I can do is
manually count rows using a `COUNT(*) OVER ...`. So I'd like to
suggest that MEASURES be part of the eventual v1 feature, if there's
no other way to determine whether a row was skipped by a previous
match. (That was less obvious to me before the fix in v17.)

--

I've been working on an implementation [1] of SQL/RPR's "parenthesized
language" and preferment order. (These are defined in SQL/Foundation
2023, section 9.41.) The tool gives you a way to figure out, for a
given pattern, what matches are supposed to be attempted and in what
order:

    $ ./src/test/modules/rpr/rpr_prefer "a b? a"
    ( ( a ( b ) ) a )
    ( ( a ( ) ) a )

Many simple patterns result in an infinite set of possible matches. So
if you use an unbounded quantifiers, you have to also use --max-rows
to limit the size of the hypothetical window frame:

    $ ./src/test/modules/rpr/rpr_prefer --max-rows 2 "^ PERMUTE(a*, b+)? $"
    ( ( ^ ( ( ( ( ( ( a ) ( b ) ) ) - ) ) ) ) $ )
    ( ( ^ ( ( ( ( ( ( ) ( b b ) ) ) - ) ) ) ) $ )
    ( ( ^ ( ( ( ( ( ( ) ( b ) ) ) - ) ) ) ) $ )
    ( ( ^ ( ( ( - ( ( ( b b ) ( ) ) ) ) ) ) ) $ )
    ( ( ^ ( ( ( - ( ( ( b ) ( a ) ) ) ) ) ) ) $ )
    ( ( ^ ( ( ( - ( ( ( b ) ( ) ) ) ) ) ) ) $ )
    ( ( ^ ( ) ) $ )

I've found this useful to check my personal understanding of the spec
and the match behavior, but it could also potentially be used to
generate test cases, or to help users debug their own patterns. For
example, a pattern that has a bunch of duplicate sequences in its PL
is probably not very well optimized:

    $ ./src/test/modules/rpr/rpr_prefer --max-rows 4 "a+ a+"
    ( ( a a a ) ( a ) )
    ( ( a a ) ( a a ) )
    ( ( a a ) ( a ) )
    ( ( a ) ( a a a ) )
    ( ( a ) ( a a ) )
    ( ( a ) ( a ) )

And patterns with catastrophic backtracking behavior tend to show a
"sawtooth" pattern in the output, with a huge number of potential
matches being generated relative to the number of rows in the frame.

My implementation is really messy -- it leaks memory like a sieve, and
I cannibalized the parser from ECPG, which just ended up as an
exercise in teaching myself flex/bison. But if there's interest in
having this kind of tool in the tree, I can work on making it
reviewable. Either way, I should be able to use it to double-check
more complicated test cases.

A while back [2], you were wondering whether our Bison implementation
would be able to parse the PATTERN grammar directly. I think this tool
proves that the answer is "yes", but PERMUTE in particular causes a
shift/reduce conflict. To fix it, I applied the same precedence
workaround that we use for CUBE and ROLLUP.

Thanks again!
--Jacob

[1] https://github.com/jchampio/postgres/tree/dev/rpr
[2] https://www.postgresql.org/message-id/20230721.151648.412762379013769790.t-ishii%40sranhm.sra.co.jp

  83. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2024-09-28T10:43:59Z

    > With some bigger partitions, I hit an `ERROR:  wrong pos: 1024`. A
> test that reproduces it is attached.

Thanks for the report. Attached is a patch on top of v22 patches to
fix the bug. We keep info in an array
(WindowAggState.reduced_frame_map) to track the rpr pattern match
result status for each row in a frame. If pattern match succeeds, the
first row in the reduced frame has status RF_FRAME_HEAD and rest of
rows have RF_SKIPPED state. A row with pattern match failure state has
RF_UNMATCHED state. Any row which is never tested has state
RF_NOT_DETERMINED. At begining the map is initialized with 1024
entries with all RF_NOT_DETERMINED state. Eventually they are replaced
with other than RF_NOT_DETERMINED state. In the error case rpr engine
tries to find 1024 th row's state in the map and failed because the
row's state has not been tested yet. I think we should treat it as
RF_NOT_DETERMINED rather than an error. Attached patch does it.

> While playing with the feature, I've been trying to identify runs of
> matched rows by eye. But it's pretty difficult -- the best I can do is
> manually count rows using a `COUNT(*) OVER ...`. So I'd like to
> suggest that MEASURES be part of the eventual v1 feature, if there's
> no other way to determine whether a row was skipped by a previous
> match. (That was less obvious to me before the fix in v17.)

I think implementing MEASURES is challenging. Especially we need to
find how our parser accepts "colname OVER
window_definition". Currently PostgreSQL's parser only accepts "func()
OVER window_definition" Even it is technically possible, I think the
v1 patch size will become much larger than now due to this.

How about inventing new window function that returns row state instead?

- match found (yes/no)
- skipped due to AFTER MATCH SKIP PAST LAST ROW (no match)

For the rest of the mail I need more time to understand. I will reply
back after studying it. For now, I just want to thank you for the
valuable information!

> --
> 
> I've been working on an implementation [1] of SQL/RPR's "parenthesized
> language" and preferment order. (These are defined in SQL/Foundation
> 2023, section 9.41.) The tool gives you a way to figure out, for a
> given pattern, what matches are supposed to be attempted and in what
> order:
> 
>     $ ./src/test/modules/rpr/rpr_prefer "a b? a"
>     ( ( a ( b ) ) a )
>     ( ( a ( ) ) a )
> 
> Many simple patterns result in an infinite set of possible matches. So
> if you use an unbounded quantifiers, you have to also use --max-rows
> to limit the size of the hypothetical window frame:
> 
>     $ ./src/test/modules/rpr/rpr_prefer --max-rows 2 "^ PERMUTE(a*, b+)? $"
>     ( ( ^ ( ( ( ( ( ( a ) ( b ) ) ) - ) ) ) ) $ )
>     ( ( ^ ( ( ( ( ( ( ) ( b b ) ) ) - ) ) ) ) $ )
>     ( ( ^ ( ( ( ( ( ( ) ( b ) ) ) - ) ) ) ) $ )
>     ( ( ^ ( ( ( - ( ( ( b b ) ( ) ) ) ) ) ) ) $ )
>     ( ( ^ ( ( ( - ( ( ( b ) ( a ) ) ) ) ) ) ) $ )
>     ( ( ^ ( ( ( - ( ( ( b ) ( ) ) ) ) ) ) ) $ )
>     ( ( ^ ( ) ) $ )
> 
> I've found this useful to check my personal understanding of the spec
> and the match behavior, but it could also potentially be used to
> generate test cases, or to help users debug their own patterns. For
> example, a pattern that has a bunch of duplicate sequences in its PL
> is probably not very well optimized:
> 
>     $ ./src/test/modules/rpr/rpr_prefer --max-rows 4 "a+ a+"
>     ( ( a a a ) ( a ) )
>     ( ( a a ) ( a a ) )
>     ( ( a a ) ( a ) )
>     ( ( a ) ( a a a ) )
>     ( ( a ) ( a a ) )
>     ( ( a ) ( a ) )
> 
> And patterns with catastrophic backtracking behavior tend to show a
> "sawtooth" pattern in the output, with a huge number of potential
> matches being generated relative to the number of rows in the frame.
> 
> My implementation is really messy -- it leaks memory like a sieve, and
> I cannibalized the parser from ECPG, which just ended up as an
> exercise in teaching myself flex/bison. But if there's interest in
> having this kind of tool in the tree, I can work on making it
> reviewable. Either way, I should be able to use it to double-check
> more complicated test cases.
> 
> A while back [2], you were wondering whether our Bison implementation
> would be able to parse the PATTERN grammar directly. I think this tool
> proves that the answer is "yes", but PERMUTE in particular causes a
> shift/reduce conflict. To fix it, I applied the same precedence
> workaround that we use for CUBE and ROLLUP.
> 
> Thanks again!
> --Jacob
> 
> [1] https://github.com/jchampio/postgres/tree/dev/rpr
> [2] https://www.postgresql.org/message-id/20230721.151648.412762379013769790.t-ishii%40sranhm.sra.co.jp

  84. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2024-09-30T00:07:51Z

    >> While playing with the feature, I've been trying to identify runs of
>> matched rows by eye. But it's pretty difficult -- the best I can do is
>> manually count rows using a `COUNT(*) OVER ...`. So I'd like to
>> suggest that MEASURES be part of the eventual v1 feature, if there's
>> no other way to determine whether a row was skipped by a previous
>> match. (That was less obvious to me before the fix in v17.)
> 
> I think implementing MEASURES is challenging. Especially we need to
> find how our parser accepts "colname OVER
> window_definition". Currently PostgreSQL's parser only accepts "func()
> OVER window_definition" Even it is technically possible, I think the
> v1 patch size will become much larger than now due to this.
> 
> How about inventing new window function that returns row state instead?
> 
> - match found (yes/no)
> - skipped due to AFTER MATCH SKIP PAST LAST ROW (no match)

Please disregard my proposal. Even if we make such a function, it
would always return NULL for unmatched rows or skipped rows, and I
think the function does not solve your problem.

However, I wonder if supporting MEASURES solves the problem either
because any columns defined by MEASURES will return NULL except the
first row in a reduced frame. Can you please show an example how to
identify runs of matched rows using MEASURES?

> For the rest of the mail I need more time to understand. I will reply
> back after studying it. For now, I just want to thank you for the
> valuable information!
> 
>> --
>> 
>> I've been working on an implementation [1] of SQL/RPR's "parenthesized
>> language" and preferment order. (These are defined in SQL/Foundation
>> 2023, section 9.41.) The tool gives you a way to figure out, for a
>> given pattern, what matches are supposed to be attempted and in what
>> order:
>> 
>>     $ ./src/test/modules/rpr/rpr_prefer "a b? a"
>>     ( ( a ( b ) ) a )
>>     ( ( a ( ) ) a )
>> 
>> Many simple patterns result in an infinite set of possible matches. So
>> if you use an unbounded quantifiers, you have to also use --max-rows
>> to limit the size of the hypothetical window frame:
>> 
>>     $ ./src/test/modules/rpr/rpr_prefer --max-rows 2 "^ PERMUTE(a*, b+)? $"
>>     ( ( ^ ( ( ( ( ( ( a ) ( b ) ) ) - ) ) ) ) $ )
>>     ( ( ^ ( ( ( ( ( ( ) ( b b ) ) ) - ) ) ) ) $ )
>>     ( ( ^ ( ( ( ( ( ( ) ( b ) ) ) - ) ) ) ) $ )
>>     ( ( ^ ( ( ( - ( ( ( b b ) ( ) ) ) ) ) ) ) $ )
>>     ( ( ^ ( ( ( - ( ( ( b ) ( a ) ) ) ) ) ) ) $ )
>>     ( ( ^ ( ( ( - ( ( ( b ) ( ) ) ) ) ) ) ) $ )
>>     ( ( ^ ( ) ) $ )

I wonder how Oracle solves the problem (an infinite set of possible
matches) without using "--max-rows" or something like that because in
my understanding Oracle supports the regular expressions and PERMUTE.

>> I've found this useful to check my personal understanding of the spec
>> and the match behavior, but it could also potentially be used to
>> generate test cases, or to help users debug their own patterns. For
>> example, a pattern that has a bunch of duplicate sequences in its PL
>> is probably not very well optimized:
>> 
>>     $ ./src/test/modules/rpr/rpr_prefer --max-rows 4 "a+ a+"
>>     ( ( a a a ) ( a ) )
>>     ( ( a a ) ( a a ) )
>>     ( ( a a ) ( a ) )
>>     ( ( a ) ( a a a ) )
>>     ( ( a ) ( a a ) )
>>     ( ( a ) ( a ) )
>> 
>> And patterns with catastrophic backtracking behavior tend to show a
>> "sawtooth" pattern in the output, with a huge number of potential
>> matches being generated relative to the number of rows in the frame.
>> 
>> My implementation is really messy -- it leaks memory like a sieve, and
>> I cannibalized the parser from ECPG, which just ended up as an
>> exercise in teaching myself flex/bison. But if there's interest in
>> having this kind of tool in the tree, I can work on making it
>> reviewable. Either way, I should be able to use it to double-check
>> more complicated test cases.

I definitely am interested in the tool!

>> A while back [2], you were wondering whether our Bison implementation
>> would be able to parse the PATTERN grammar directly. I think this tool
>> proves that the answer is "yes", but PERMUTE in particular causes a
>> shift/reduce conflict. To fix it, I applied the same precedence
>> workaround that we use for CUBE and ROLLUP.

That's a good news!

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  85. Re: Row pattern recognition

    Jacob Champion <jacob.champion@enterprisedb.com> — 2024-10-01T12:48:02Z

    On Sun, Sep 29, 2024 at 5:08 PM Tatsuo Ishii <ishii@postgresql.org> wrote:
> > I think implementing MEASURES is challenging. Especially we need to
> > find how our parser accepts "colname OVER
> > window_definition". Currently PostgreSQL's parser only accepts "func()
> > OVER window_definition" Even it is technically possible, I think the
> > v1 patch size will become much larger than now due to this.

[resending, to the whole list this time]

Yeah. In any case, I'm not the right person to bolt MEASURES onto the
existing grammar... my misadventures in the PATTERN parser have
highlighted how little I know about Bison. :D

> Please disregard my proposal. Even if we make such a function, it
> would always return NULL for unmatched rows or skipped rows, and I
> think the function does not solve your problem.
>
> However, I wonder if supporting MEASURES solves the problem either
> because any columns defined by MEASURES will return NULL except the
> first row in a reduced frame. Can you please show an example how to
> identify runs of matched rows using MEASURES?

I think you're probably right; my suggestion can't distinguish between
skipped (but previously matched) rows and entirely-unmatched rows. The
test case I'd been working with returned an empty match as a fallback,
so it wouldn't have had that problem in practice. I was hoping that
one of the existing whole-partition window functions would allow me to
cobble something together based on the COUNT(*) measure, but after
searching for a while I haven't been able to come up with a solution.

Maybe it's just too niche for the window-function version of this --
after all, it only makes sense when using both INITIAL and AFTER MATCH
SKIP PAST LAST ROW. A more general solution could identify the
row_number of the first and last rows of the window frame, perhaps?
But a frame isn't guaranteed to be contiguous, so maybe that doesn't
make sense either. Ugh.

> I wonder how Oracle solves the problem (an infinite set of possible
> matches) without using "--max-rows" or something like that because in
> my understanding Oracle supports the regular expressions and PERMUTE.

I chose a confusing way to describe it, sorry. The parenthesized
language for a pattern can be an infinite set, because A+ could match
"( A )" or "( A A )" or "( A A A )" and so on forever. But that
doesn't apply to our regex engine in practice; our tables have a
finite number of rows, and I *think* the PL for a finite number of
rows is also finite, due to the complicated rules on where empty
matches are allowed to appear in the language. (In any case, my tool
doesn't guard against infinite recursion...)

> >> My implementation is really messy -- it leaks memory like a sieve, and
> >> I cannibalized the parser from ECPG, which just ended up as an
> >> exercise in teaching myself flex/bison. But if there's interest in
> >> having this kind of tool in the tree, I can work on making it
> >> reviewable. Either way, I should be able to use it to double-check
> >> more complicated test cases.
>
> I definitely am interested in the tool!

Okay, good to know! I will need to clean it up considerably, and
figure out whether I've duplicated more code than I should have.

> >> A while back [2], you were wondering whether our Bison implementation
> >> would be able to parse the PATTERN grammar directly. I think this tool
> >> proves that the answer is "yes", but PERMUTE in particular causes a
> >> shift/reduce conflict. To fix it, I applied the same precedence
> >> workaround that we use for CUBE and ROLLUP.
>
> That's a good news!

+1

Thanks,
--Jacob

  86. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2024-10-22T04:53:43Z

    Hi,

I wonder how "PREV(col + 1)" is different from "PREV(col) + 1".
Currently my RPR implementation does not allow PREV(col + 1). If
"PREV(col + 1)" is different from "PREV(col) + 1", it maybe worthwhile
to implement "PREV(col + 1)".

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  87. Row pattern recognition

    David G. Johnston <david.g.johnston@gmail.com> — 2024-10-22T05:29:34Z

    On Monday, October 21, 2024, Tatsuo Ishii <ishii@postgresql.org> wrote:
>
> I wonder how "PREV(col + 1)" is different from "PREV(col) + 1".
> Currently my RPR implementation does not allow PREV(col + 1). If
> "PREV(col + 1)" is different from "PREV(col) + 1", it maybe worthwhile
> to implement "PREV(col + 1)".
>

Interesting feature that I’m now just seeing.

The expression PREV(column_name) produces a value output taken from the
given named column in the preceding frame row.  It doesn’t make any sense
to me to attempt to add the integer 1 to an identifier that is being used
as a value input to a “function”.  It would also seem quite odd if “+ 1”
had something to do with row selection as opposed to simply being an
operator “+(column_name%type, integer)” expression.

Maybe RPR is defining something special here I haven't yet picked up on, in
which case just ignore this.  But if I read: “UP as price > prev(price +
1)” in the opening example it would be quite non-intuitive to reason out
the meaning.  “Price > prev(price) + 1” would mean my current row is at
least one (e.g. dollar per share) more than the value of the previous
period.

David J.

  88. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2024-10-22T10:19:09Z

    > On Monday, October 21, 2024, Tatsuo Ishii <ishii@postgresql.org> wrote:
>>
>> I wonder how "PREV(col + 1)" is different from "PREV(col) + 1".
>> Currently my RPR implementation does not allow PREV(col + 1). If
>> "PREV(col + 1)" is different from "PREV(col) + 1", it maybe worthwhile
>> to implement "PREV(col + 1)".
>>
> 
> Interesting feature that I’m now just seeing.
> 
> The expression PREV(column_name) produces a value output taken from the
> given named column in the preceding frame row.  It doesn’t make any sense
> to me to attempt to add the integer 1 to an identifier that is being used
> as a value input to a “function”.  It would also seem quite odd if “+ 1”
> had something to do with row selection as opposed to simply being an
> operator “+(column_name%type, integer)” expression.

According to the ISO/IEC 9075-2:2016, 6.26 <row pattern navigation
operation> (I don't have access to SQL 2023) PREV (and NEXT) is
defined as:

<row pattern navigation: physical> ::=
 <prev or next> <left paren> <value expression> [ <comma> <physical offset> ] <right paren>

(Besides <row pattern navigation: physical>, there are <row pattern
navigation: logical> and <row pattern navigation: compound> but I
ignore them here).

So PREV's first argument is a value expression (VE). VE shall contain
at least one row pattern column reference. <set function
specification>, <window function specification> or <row patter
navigation operation> are not permitted.

From this, I don't see any reason PREV(column_name + 1) is prohibited
unless I miss something.

I think even PREV(column_name1 + column_name2) is possible. I see
similar example in ISO/IEC 19075-5:2021, 5.6.2 "PREV and NEXT".

> Maybe RPR is defining something special here I haven't yet picked up on, in
> which case just ignore this.  But if I read: “UP as price > prev(price +
> 1)” in the opening example it would be quite non-intuitive to reason out
> the meaning.  “Price > prev(price) + 1” would mean my current row is at
> least one (e.g. dollar per share) more than the value of the previous
> period.

Acording to ISO/IEC 9075-2:2016 "4.21.2 Row pattern navigation operations",

  <row pattern navigation operation> evaluates a <value expression> VE
  in a row NR, which may be different than current row CR.

From this I think PREV(col + 1) should be interpreted as:

1. go to the previous row.
2. evaluate "col + 1" at the current row (that was previous row).
3. return the result.

If my understanding is correct, prev(price + 1) has the same meaning
as prev(price) + 1.

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  89. Re: Row pattern recognition

    Vik Fearing <vik@postgresfriends.org> — 2024-10-22T13:12:43Z

    On 22/10/2024 12:19, Tatsuo Ishii wrote:
> Acording to ISO/IEC 9075-2:2016 "4.21.2 Row pattern navigation operations",
>
>    <row pattern navigation operation> evaluates a <value expression> VE
>    in a row NR, which may be different than current row CR.
>
>  From this I think PREV(col + 1) should be interpreted as:
>
> 1. go to the previous row.
> 2. evaluate "col + 1" at the current row (that was previous row).
> 3. return the result.
>
> If my understanding is correct, prev(price + 1) has the same meaning
> as prev(price) + 1.



This is how I read the specification also.

-- 

Vik Fearing

  90. Re: Row pattern recognition

    David G. Johnston <david.g.johnston@gmail.com> — 2024-10-22T14:19:41Z

    On Tue, Oct 22, 2024 at 6:12 AM Vik Fearing <vik@postgresfriends.org> wrote:

>
> On 22/10/2024 12:19, Tatsuo Ishii wrote:
>
> Acording to ISO/IEC 9075-2:2016 "4.21.2 Row pattern navigation operations",
>
>   <row pattern navigation operation> evaluates a <value expression> VE
>   in a row NR, which may be different than current row CR.
>
> From this I think PREV(col + 1) should be interpreted as:
>
> 1. go to the previous row.
> 2. evaluate "col + 1" at the current row (that was previous row).
> 3. return the result.
>
> If my understanding is correct, prev(price + 1) has the same meaning
> as prev(price) + 1.
>
>
>
> This is how I read the specification also.
>
>
>
That makes sense.  Definitely much nicer to only have to write PREV once if
the expression you are evaluating involves multiple columns.  And is also
consistent with window function "value" behavior.

Thanks!

David J.

  91. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2024-10-25T04:04:53Z

    > On Tue, Oct 22, 2024 at 6:12 AM Vik Fearing <vik@postgresfriends.org> wrote:
> 
>>
>> On 22/10/2024 12:19, Tatsuo Ishii wrote:
>>
>> Acording to ISO/IEC 9075-2:2016 "4.21.2 Row pattern navigation operations",
>>
>>   <row pattern navigation operation> evaluates a <value expression> VE
>>   in a row NR, which may be different than current row CR.
>>
>> From this I think PREV(col + 1) should be interpreted as:
>>
>> 1. go to the previous row.
>> 2. evaluate "col + 1" at the current row (that was previous row).
>> 3. return the result.
>>
>> If my understanding is correct, prev(price + 1) has the same meaning
>> as prev(price) + 1.
>>
>>
>>
>> This is how I read the specification also.
>>
>>
>>
> That makes sense.  Definitely much nicer to only have to write PREV once if
> the expression you are evaluating involves multiple columns.  And is also
> consistent with window function "value" behavior.

Thanks to all who joined the discussion. I decided to support PREV and
NEXT in my RPR patches to allow to have multiple columns and other
expressions in their argument. e.g.

CREATE TEMP TABLE rpr1 (id INTEGER, i SERIAL, j INTEGER);
INSERT INTO rpr1(id, j) SELECT 1, g*2 FROM generate_series(1, 10) AS g;
SELECT id, i, j, count(*) OVER w
 FROM rpr1
 WINDOW w AS (
 PARTITION BY id
 ORDER BY i
 ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
 AFTER MATCH SKIP PAST LAST ROW
 INITIAL
 PATTERN (START COND+)
 DEFINE
  START AS TRUE,
  COND AS PREV(i + j + 1) < 10
);
 id | i  | j  | count 
----+----+----+-------
  1 |  1 |  2 |     3
  1 |  2 |  4 |     0
  1 |  3 |  6 |     0
  1 |  4 |  8 |     0
  1 |  5 | 10 |     0
  1 |  6 | 12 |     0
  1 |  7 | 14 |     0
  1 |  8 | 16 |     0
  1 |  9 | 18 |     0
  1 | 10 | 20 |     0
(10 rows)

Attached are the v23 patches. V23 also includes the fix for the problem
pointed out by Jacob Champion and test cases from him. Thank you, Jacob.
https://www.postgresql.org/message-id/CAOYmi%2Bns3kHjC83ap_BCfJCL0wfO5BJ_sEByOEpgNOrsPhqQTg%40mail.gmail.com

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  92. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2024-12-19T06:19:50Z

    I have looked into the performance of current RPR implementation,
especially when the number of rows in a reduced frame is large (like
over 10k). Below is a simple benchmark against pgbench database. The
SQL will produce a reduced frame having 10k rows.

EXPLAIN (ANALYZE)
SELECT aid, bid, count(*) OVER w
FROM pgbench_accounts WHERE aid <= 10000
WINDOW w AS (
PARTITION BY bid
ORDER BY aid
ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
AFTER MATCH SKIP PAST LAST ROW
INITIAL
PATTERN (START UP+)
DEFINE
START AS TRUE,
UP AS aid > PREV(aid)
);

This took 722 ms on my laptop. It's not very quick. Moreover, if I
expand the reduced frame size to 100k (aid <= 100000), OOM killer
triggered. I looked into the code and found that do_pattern_match in
nodeWindowAgg.c is one of the major problems. It calls regexp_instr to
know whether the regular expression derived from a PATTERN clause
(e.g. "ab+c+") matches an encoded row pattern variable string
(e.g. "abbcc"). The latter string could be quite long: the length
could be as same as the number of rows in the reduced frame. Thus, The
length could become 100k if the frame size is 100k. Unfortunately
regexp_instr needs to allocate and convert the input string to wchar
(it's 4-byte long for each character), which uses 4x space bigger than
the original input string. In RPR case the input string is always
ASCII and does not need to be converted to wchar. So I decided to
switch to the standard regular expression engine coming with OS. With
this change, I got 2x speed up in the 10k case.

v23 patch: 722.618 ms (average of 3 runs)
new patch: 322.5913 ms (average of 3 runs)

Also I tried the 100k rows reduced frame case. It was slow (took 26
seconds) but it completed without OOM killer.  Attached is the
patch. The change was in 0005 only. Other patches were not changed
from v23.

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  93. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2024-12-21T09:20:04Z

    > I have looked into the performance of current RPR implementation,
> especially when the number of rows in a reduced frame is large (like
> over 10k). Below is a simple benchmark against pgbench database. The
> SQL will produce a reduced frame having 10k rows.
> 
> EXPLAIN (ANALYZE)
> SELECT aid, bid, count(*) OVER w
> FROM pgbench_accounts WHERE aid <= 10000
> WINDOW w AS (
> PARTITION BY bid
> ORDER BY aid
> ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
> AFTER MATCH SKIP PAST LAST ROW
> INITIAL
> PATTERN (START UP+)
> DEFINE
> START AS TRUE,
> UP AS aid > PREV(aid)
> );
> 
> This took 722 ms on my laptop. It's not very quick. Moreover, if I
> expand the reduced frame size to 100k (aid <= 100000), OOM killer
> triggered. I looked into the code and found that do_pattern_match in
> nodeWindowAgg.c is one of the major problems. It calls regexp_instr to
> know whether the regular expression derived from a PATTERN clause
> (e.g. "ab+c+") matches an encoded row pattern variable string
> (e.g. "abbcc"). The latter string could be quite long: the length
> could be as same as the number of rows in the reduced frame. Thus, The
> length could become 100k if the frame size is 100k. Unfortunately
> regexp_instr needs to allocate and convert the input string to wchar
> (it's 4-byte long for each character), which uses 4x space bigger than
> the original input string. In RPR case the input string is always
> ASCII and does not need to be converted to wchar. So I decided to
> switch to the standard regular expression engine coming with OS. With
> this change, I got 2x speed up in the 10k case.
> 
> v23 patch: 722.618 ms (average of 3 runs)
> new patch: 322.5913 ms (average of 3 runs)
> 
> Also I tried the 100k rows reduced frame case. It was slow (took 26
> seconds) but it completed without OOM killer.  Attached is the
> patch. The change was in 0005 only. Other patches were not changed
> from v23.

The CFBot starts complaining about the patch. It fails in Windows
environment test because regex.h does not exist. I concluded that on
Windows it's not a good idea to use the standard regexp library (I am
not familiar with Windows. If my thought is not correct, please let me
know). So I switched to the PostgreSQL's builtin core regexp
library. Although the interface requires to use pg_wchar which spends
4x memory comparing with the standard regexp (that's why I wanted to
avoid using it), the result seems to be not so bad. It consumes only
10MB or so more memory when processing 100k rows in a frame. Good news
is, it runs slightly faster than the standard regexp (19 vs. 26
seconds).

Attached is the v25 patch to use the PostgreSQL's regexp library.
Most changes are in nodeWindowAgg.c, which is in the 5th patch.

In the patches I also fixed some memory leaks and run pgindent with
updated typedefs.list. Now the patch includes a patch for
typedefs.list (the 8th patch).

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  94. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2024-12-30T13:37:18Z

    I have added further optimization to the v25 patch.

While generating possible input strings that may satisfy the pattern
string, it is possible to omit to run regexp in some cases. Since
regexp matching is heavy operation, especially if it is applied to
long string, it is beneficial for RPR to reduce the number of regexp
runs.

If the tail pattern variable has '+' quantifier and previously the
input string was confirmed to be matched the pattern string, and the
same character as the tail pattern string is added, we don't need run
regexp match again because the new input string surely matches the
pattern string. Suppose a pattern string is "ab+" and the current
input string is "ab" (this satisfies "ab+"). If the new input string
is "abb", then "abb" surely matches "ab+" too and we don't need to run
regexp again.

In v26 patch, by using the technique above I get performance
improvement.

>> EXPLAIN (ANALYZE)
>> SELECT aid, bid, count(*) OVER w
>> FROM pgbench_accounts WHERE aid <= 10000
>> WINDOW w AS (
>> PARTITION BY bid
>> ORDER BY aid
>> ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>> AFTER MATCH SKIP PAST LAST ROW
>> INITIAL
>> PATTERN (START UP+)
>> DEFINE
>> START AS TRUE,
>> UP AS aid > PREV(aid)
>> );

This SQL took 322.5913 ms (average in 3 runs) in v24. With v26 patch,
it takes 41.84 ms, which is over 7 times improvement. Also I run the
SQL in 100k row case. v23 took 26 seconds. With the v26 patch it takes
1195.603 ms, which is over 21 times improvement.

I think a pattern string ended up with '+' is one of common use cases
of RPR, and I believe the improvement is useful for many RPR
applications.

I also add following changes to v25.

- Fix do_pattern_match to use the top memory context to store compiled
  re cache. Before it was in per query memory context. This causes a
  trouble because do_pattern_match checks the cache existence using
  a static variable.

- Refactor search_str_set, which is a workhorse of pattern matching,
  into multiple functions to understand the logic easier.
  
Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  95. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2024-12-30T23:57:07Z

    > I have added further optimization to the v25 patch.
> 
> While generating possible input strings that may satisfy the pattern
> string, it is possible to omit to run regexp in some cases. Since
> regexp matching is heavy operation, especially if it is applied to
> long string, it is beneficial for RPR to reduce the number of regexp
> runs.
> 
> If the tail pattern variable has '+' quantifier and previously the
> input string was confirmed to be matched the pattern string, and the
> same character as the tail pattern string is added, we don't need run
> regexp match again because the new input string surely matches the
> pattern string. Suppose a pattern string is "ab+" and the current
> input string is "ab" (this satisfies "ab+"). If the new input string
> is "abb", then "abb" surely matches "ab+" too and we don't need to run
> regexp again.
> 
> In v26 patch, by using the technique above I get performance
> improvement.
> 
>>> EXPLAIN (ANALYZE)
>>> SELECT aid, bid, count(*) OVER w
>>> FROM pgbench_accounts WHERE aid <= 10000
>>> WINDOW w AS (
>>> PARTITION BY bid
>>> ORDER BY aid
>>> ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>>> AFTER MATCH SKIP PAST LAST ROW
>>> INITIAL
>>> PATTERN (START UP+)
>>> DEFINE
>>> START AS TRUE,
>>> UP AS aid > PREV(aid)
>>> );
> 
> This SQL took 322.5913 ms (average in 3 runs) in v24. With v26 patch,
> it takes 41.84 ms, which is over 7 times improvement. Also I run the
> SQL in 100k row case. v23 took 26 seconds. With the v26 patch it takes
> 1195.603 ms, which is over 21 times improvement.
> 
> I think a pattern string ended up with '+' is one of common use cases
> of RPR, and I believe the improvement is useful for many RPR
> applications.
> 
> I also add following changes to v25.
> 
> - Fix do_pattern_match to use the top memory context to store compiled
>   re cache. Before it was in per query memory context. This causes a
>   trouble because do_pattern_match checks the cache existence using
>   a static variable.
> 
> - Refactor search_str_set, which is a workhorse of pattern matching,
>   into multiple functions to understand the logic easier.

CFBot complains a compiler error in the v26 patch.
Attached is v27 patch to fix this. Also some typo in comment are fixed.

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  96. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2025-01-11T05:46:11Z

    Attached are the v28 patches to implement a subset of Row Pattern
Recognition feature defined in the SQL standard.

In this patch set:

- Reduce the patch size. Comparing with v27, the patch size is trimmed
  from 5296 lines down to 5073 lines. This is mainly achieved in the
  raw parser patch so that non implemented features are removed from
  the grammar file.

- Use newly introduced makeStringInfoExt() instead of makeStringInfo()
  in nodeWindowAgg.c, which removes a few unnecessary codes and should
  give slightly better performance.

- Fix bugs in the doc and add it more description regarding RPR
  syntax.

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  97. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2025-03-13T12:51:29Z

    Attached are the v29 patch sets to implement the subset of row pattern
recognition feature defined in the SQL standard.

In this patch set:

- Adjust 0003 and 0004 to deal with the recent changes made by 8b1b342.
 
- Add tests to 0007 for CREATE VIEW/pg_get_viewdef.

Best reagards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  98. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2025-04-22T23:55:33Z

    Attached are the v30 patches, just adding a patch to change the
default io_method parameter to sync, expecting this affects something
to the recent CFbot failure.
https://commitfest.postgresql.org/patch/4460/
https://cirrus-ci.com/task/6078653164945408
which is similar to:
https://www.postgresql.org/message-id/20250422.111139.1502127917165838403.ishii%40postgresql.org

(Once the issue resolved, the patch should be removed, of course)

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  99. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2025-04-23T01:07:29Z

    > Attached are the v30 patches, just adding a patch to change the
> default io_method parameter to sync, expecting this affects something
> to the recent CFbot failure.
> https://commitfest.postgresql.org/patch/4460/
> https://cirrus-ci.com/task/6078653164945408
> which is similar to:
> https://www.postgresql.org/message-id/20250422.111139.1502127917165838403.ishii%40postgresql.org

CFBot has just run tests against v30 patches and now it turns to green
again!  I guess io_method = sync definitely did the trick.  Note that
previously only the Windows Server 2019, VS 2019 - Neason & ninja test
had failed.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  100. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2025-05-06T00:53:51Z

    > Attached are the v30 patches, just adding a patch to change the
> default io_method parameter to sync, expecting this affects something
> to the recent CFbot failure.
> https://commitfest.postgresql.org/patch/4460/
> https://cirrus-ci.com/task/6078653164945408
> which is similar to:
> https://www.postgresql.org/message-id/20250422.111139.1502127917165838403.ishii%40postgresql.org
> 
> (Once the issue resolved, the patch should be removed, of course)

It seems this has turned to green since May 2, 2025.
https://commitfest.postgresql.org/patch/5679/.

The last time it turned to red was April 16, 2025. Maybe something
committed between the period solved the cause of red, but I don't know
exactly which commit.

Anyway v31 patches now remove the change to default io_method
parameter to see if it passes Windows Server 2019, VS 2019 - Meson &
ninja test.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  101. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2025-05-06T07:28:42Z

    >> Attached are the v30 patches, just adding a patch to change the
>> default io_method parameter to sync, expecting this affects something
>> to the recent CFbot failure.
>> https://commitfest.postgresql.org/patch/4460/
>> https://cirrus-ci.com/task/6078653164945408
>> which is similar to:
>> https://www.postgresql.org/message-id/20250422.111139.1502127917165838403.ishii%40postgresql.org
>> 
>> (Once the issue resolved, the patch should be removed, of course)
> 
> It seems this has turned to green since May 2, 2025.
> https://commitfest.postgresql.org/patch/5679/.
> 
> The last time it turned to red was April 16, 2025. Maybe something
> committed between the period solved the cause of red, but I don't know
> exactly which commit.
> 
> Anyway v31 patches now remove the change to default io_method
> parameter to see if it passes Windows Server 2019, VS 2019 - Meson &
> ninja test.

Now I see it passes the test.
https://cirrus-ci.com/build/5671612202090496

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  102. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2025-08-16T08:45:52Z

    Attached are the v32 patches for Row pattern recognition.  Recent
changes to doc/src/sgml/func.sgml required v31 to be rebased. Other
than that, nothing has been changed.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  103. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2025-09-24T10:35:23Z

    Attached are the v33 patches for Row pattern recognition.  The
difference from v32 is that the raw parse tree printing patch is not
included in v33. This is because now that we have
debug_print_raw_parse.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  104. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2025-11-17T06:57:15Z

    Attached are the v34 patches for Row pattern recognition.
Notihing has been changed. Just rebased.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  105. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2025-11-18T02:33:20Z

    Attached are the v35 patches for Row pattern recognition.  In v34-0001
gram.y patch, %type for RPR were misplaced. v35-0001 fixes this. Other
patches are not changed.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  106. Re: Row pattern recognition

    Chao Li <li.evan.chao@gmail.com> — 2025-11-18T05:03:08Z

    Hi Tatsuo-san,

I just reviewed 0006 docs changes and 0001. I plan to slowly review the patch, maybe one commit a day. 

> On Nov 18, 2025, at 10:33, Tatsuo Ishii <ishii@postgresql.org> wrote:
> 
> Attached are the v35 patches for Row pattern recognition.  In v34-0001
> gram.y patch, %type for RPR were misplaced. v35-0001 fixes this. Other
> patches are not changed.
> 
> Best regards,
> --
> Tatsuo Ishii
> SRA OSS K.K.
> English: http://www.sraoss.co.jp/index_en/
> Japanese:http://www.sraoss.co.jp
> <v35-0001-Row-pattern-recognition-patch-for-raw-parser.patch><v35-0002-Row-pattern-recognition-patch-parse-analysis.patch><v35-0003-Row-pattern-recognition-patch-rewriter.patch><v35-0004-Row-pattern-recognition-patch-planner.patch><v35-0005-Row-pattern-recognition-patch-executor.patch><v35-0006-Row-pattern-recognition-patch-docs.patch><v35-0007-Row-pattern-recognition-patch-tests.patch><v35-0008-Row-pattern-recognition-patch-typedefs.list.patch>

I got a few comments, maybe just questions:

1 - 0001 - kwlist.h
```
+PG_KEYWORD("define", DEFINE, RESERVED_KEYWORD, BARE_LABEL)
```

Why do we add “define” as a reserved keyword? From the SQL example you put in 0006:
```
<programlisting>
SELECT company, tdate, price,
 first_value(price) OVER w,
 max(price) OVER w,
 count(price) OVER w
FROM stock
 WINDOW w AS (
 PARTITION BY company
 ORDER BY tdate
 ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
 AFTER MATCH SKIP PAST LAST ROW
 INITIAL
 PATTERN (LOWPRICE UP+ DOWN+)
 DEFINE
  LOWPRICE AS price &lt;= 100,
  UP AS price &gt; PREV(price),
  DOWN AS price &lt; PREV(price)
);
</programlisting>
```

PARTITION is at the same level as DEFINE, but it’s not defined as a reserved keyword:
```
PG_KEYWORD("partition", PARTITION, UNRESERVED_KEYWORD, BARE_LABEL)
```

Even in this patch,”initial”,”past”, “pattern” and “seek” are defined as unreserved, why?  

So I just want to clarify.

2 - 0001 - gram.y
```
opt_row_pattern_initial_or_seek:
			INITIAL		{ $$ = true; }
			| SEEK
				{
					ereport(ERROR,
							(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
							 errmsg("SEEK is not supported"),
							 errhint("Use INITIAL instead."),
							 parser_errposition(@1)));
				}
			| /*EMPTY*/		{ $$ = true; }
		;
```

As SEEK is specially listed here, I guess it might be supported in future. If that is true, would it be better to defer the semantic check to later parse phase, which would future work easier.

3 - 0001 - parsenodes.h
```
+/*
+ * RowPatternCommonSyntax - raw representation of row pattern common syntax
+ *
+ */
+typedef struct RPCommonSyntax
+{
+	NodeTag		type;
+	RPSkipTo	rpSkipTo;		/* Row Pattern AFTER MATCH SKIP type */
+	bool		initial;		/* true if <row pattern initial or seek> is
+								 * initial */
+	List	   *rpPatterns;		/* PATTERN variables (list of A_Expr) */
+	List	   *rpDefs;			/* row pattern definitions clause (list of
+								 * ResTarget) */
+} RPCommonSyntax;
+
 /*
  * WindowDef - raw representation of WINDOW and OVER clauses
  *
@@ -593,6 +618,7 @@ typedef struct WindowDef
 	char	   *refname;		/* referenced window name, if any */
 	List	   *partitionClause;	/* PARTITION BY expression list */
 	List	   *orderClause;	/* ORDER BY (list of SortBy) */
+	RPCommonSyntax *rpCommonSyntax; /* row pattern common syntax */
```

RP fields are directly defined in WindowClause, then why do we need a wrapper of RPCommonSyntax? Can we directly define RP fields in WindowRef as well?

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

  107. Re: Row pattern recognition

    Chao Li <li.evan.chao@gmail.com> — 2025-11-18T05:05:52Z

    
> On Nov 18, 2025, at 13:03, Chao Li <li.evan.chao@gmail.com> wrote:
> 
> Hi Tatsuo-san,
> 
> I just reviewed 0006 docs changes and 0001. I plan to slowly review the patch, maybe one commit a day. 
> 
>> On Nov 18, 2025, at 10:33, Tatsuo Ishii <ishii@postgresql.org> wrote:
>> 
>> Attached are the v35 patches for Row pattern recognition.  In v34-0001
>> gram.y patch, %type for RPR were misplaced. v35-0001 fixes this. Other
>> patches are not changed.
>> 
>> Best regards,
>> --
>> Tatsuo Ishii
>> SRA OSS K.K.
>> English: http://www.sraoss.co.jp/index_en/
>> Japanese:http://www.sraoss.co.jp
>> <v35-0001-Row-pattern-recognition-patch-for-raw-parser.patch><v35-0002-Row-pattern-recognition-patch-parse-analysis.patch><v35-0003-Row-pattern-recognition-patch-rewriter.patch><v35-0004-Row-pattern-recognition-patch-planner.patch><v35-0005-Row-pattern-recognition-patch-executor.patch><v35-0006-Row-pattern-recognition-patch-docs.patch><v35-0007-Row-pattern-recognition-patch-tests.patch><v35-0008-Row-pattern-recognition-patch-typedefs.list.patch>
> 
> I got a few comments, maybe just questions:
> 
> 1 - 0001 - kwlist.h
> ```
> +PG_KEYWORD("define", DEFINE, RESERVED_KEYWORD, BARE_LABEL)
> ```
> 
> Why do we add “define” as a reserved keyword? From the SQL example you put in 0006:
> ```
> <programlisting>
> SELECT company, tdate, price,
> first_value(price) OVER w,
> max(price) OVER w,
> count(price) OVER w
> FROM stock
> WINDOW w AS (
> PARTITION BY company
> ORDER BY tdate
> ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
> AFTER MATCH SKIP PAST LAST ROW
> INITIAL
> PATTERN (LOWPRICE UP+ DOWN+)
> DEFINE
>  LOWPRICE AS price &lt;= 100,
>  UP AS price &gt; PREV(price),
>  DOWN AS price &lt; PREV(price)
> );
> </programlisting>
> ```
> 
> PARTITION is at the same level as DEFINE, but it’s not defined as a reserved keyword:
> ```
> PG_KEYWORD("partition", PARTITION, UNRESERVED_KEYWORD, BARE_LABEL)
> ```
> 
> Even in this patch,”initial”,”past”, “pattern” and “seek” are defined as unreserved, why?  
> 
> So I just want to clarify.
> 
> 2 - 0001 - gram.y
> ```
> opt_row_pattern_initial_or_seek:
> INITIAL { $$ = true; }
> | SEEK
> {
> ereport(ERROR,
> (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
> errmsg("SEEK is not supported"),
> errhint("Use INITIAL instead."),
> parser_errposition(@1)));
> }
> | /*EMPTY*/ { $$ = true; }
> ;
> ```
> 
> As SEEK is specially listed here, I guess it might be supported in future. If that is true, would it be better to defer the semantic check to later parse phase, which would future work easier.
> 
> 3 - 0001 - parsenodes.h
> ```
> +/*
> + * RowPatternCommonSyntax - raw representation of row pattern common syntax
> + *
> + */
> +typedef struct RPCommonSyntax
> +{
> + NodeTag type;
> + RPSkipTo rpSkipTo; /* Row Pattern AFTER MATCH SKIP type */
> + bool initial; /* true if <row pattern initial or seek> is
> + * initial */
> + List   *rpPatterns; /* PATTERN variables (list of A_Expr) */
> + List   *rpDefs; /* row pattern definitions clause (list of
> + * ResTarget) */
> +} RPCommonSyntax;
> +
> /*
>  * WindowDef - raw representation of WINDOW and OVER clauses
>  *
> @@ -593,6 +618,7 @@ typedef struct WindowDef
> char   *refname; /* referenced window name, if any */
> List   *partitionClause; /* PARTITION BY expression list */
> List   *orderClause; /* ORDER BY (list of SortBy) */
> + RPCommonSyntax *rpCommonSyntax; /* row pattern common syntax */
> ```
> 
> RP fields are directly defined in WindowClause, then why do we need a wrapper of RPCommonSyntax? Can we directly define RP fields in WindowRef as well?
> 

4 - 0001 - parsenodes.h
```
+	/* Row Pattern AFTER MACH SKIP clause */
```

Typo: MACH -> MATCH

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

  108. Re: Row pattern recognition

    Vik Fearing <vik@postgresfriends.org> — 2025-11-18T11:19:46Z

    On 18/11/2025 06:03, Chao Li wrote:
> 1 - 0001 - kwlist.h
> ```
> +PG_KEYWORD("define", DEFINE, RESERVED_KEYWORD, BARE_LABEL)
> ```
>
> Why do we add “define” as a reserved keyword? From the SQL example you put in 0006:
> ```
> <programlisting>
> SELECT company, tdate, price,
>   first_value(price) OVER w,
>   max(price) OVER w,
>   count(price) OVER w
> FROM stock
>   WINDOW w AS (
>   PARTITION BY company
>   ORDER BY tdate
>   ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>   AFTER MATCH SKIP PAST LAST ROW
>   INITIAL
>   PATTERN (LOWPRICE UP+ DOWN+)
>   DEFINE
>    LOWPRICE AS price &lt;= 100,
>    UP AS price &gt; PREV(price),
>    DOWN AS price &lt; PREV(price)
> );
> </programlisting>
> ```
>
> PARTITION is at the same level as DEFINE, but it’s not defined as a reserved keyword:
> ```
> PG_KEYWORD("partition", PARTITION, UNRESERVED_KEYWORD, BARE_LABEL)
> ```
>
> Even in this patch,”initial”,”past”, “pattern” and “seek” are defined as unreserved, why?
>
> So I just want to clarify.


Because of position. Without making DEFINE a reserved keyword, how do 
you know that it isn't another variable in the PATTERN clause?

-- 

Vik Fearing

  109. Re: Row pattern recognition

    Chao Li <li.evan.chao@gmail.com> — 2025-11-19T04:14:03Z

    
> On Nov 18, 2025, at 19:19, Vik Fearing <vik@postgresfriends.org> wrote:
> 
> 
> Because of position. Without making DEFINE a reserved keyword, how do you know that it isn't another variable in the PATTERN clause?
> 

Ah, thanks for the clarification. Now I got it.

I’m continue to review 0002.

5 - 0002 - parse_clause.c
```
+		ereport(ERROR,
+				(errcode(ERRCODE_SYNTAX_ERROR),
+				 errmsg("FRAME must start at current row when row patttern recognition is used")));
```

Nit typo: patttern -> pattern

6 - 0002 - parse_clause.c
```
+	/* DEFINE variable name initials */
+	static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz”;
```

This string can also be const, so “static const char *”

7 - 0002 - parse_clause.c
```
+	/*
+	 * Create list of row pattern DEFINE variable name's initial. We assign
+	 * [a-z] to them (up to 26 variable names are allowed).
+	 */
+	restargets = NIL;
+	i = 0;
+	initialLen = strlen(defineVariableInitials);
+
+	foreach(lc, windef->rpCommonSyntax->rpDefs)
+	{
+		char		initial[2];
+
+		restarget = (ResTarget *) lfirst(lc);
+		name = restarget->name;
```

Looks like “name” is not used inside “foreach”.

8 - 0002 - parse_clause.c
```
+	/*
+	 * Create list of row pattern DEFINE variable name's initial. We assign
+	 * [a-z] to them (up to 26 variable names are allowed).
+	 */
+	restargets = NIL;
+	i = 0;
+	initialLen = strlen(defineVariableInitials);
+
+	foreach(lc, windef->rpCommonSyntax->rpDefs)
+	{
+		char		initial[2];
```

I guess this “foreach” for build initial list can be combined into the previous foreach loop of checking duplication. If an def has no dup, then assign an initial to it.

9 - 0002 - parse_clause.c
```
+static void
+transformPatternClause(ParseState *pstate, WindowClause *wc,
+					   WindowDef *windef)
+{
+	ListCell   *lc;
+
+	/*
+	 * Row Pattern Common Syntax clause exists?
+	 */
+	if (windef->rpCommonSyntax == NULL)
+		return;
```

Checking  (windef->rpCommonSyntax == NULL) seems a duplicate, because transformPatternClause() is only called by transformRPR(), and transformRPR() has done the check.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

  110. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2025-11-19T06:51:02Z

    > On 18/11/2025 06:03, Chao Li wrote:
>> 1 - 0001 - kwlist.h
>> ```
>> +PG_KEYWORD("define", DEFINE, RESERVED_KEYWORD, BARE_LABEL)
>> ```
>>
>> Why do we add “define” as a reserved keyword? From the SQL example
>> you put in 0006:
>> ```
>> <programlisting>
>> SELECT company, tdate, price,
>>   first_value(price) OVER w,
>>   max(price) OVER w,
>>   count(price) OVER w
>> FROM stock
>>   WINDOW w AS (
>>   PARTITION BY company
>>   ORDER BY tdate
>>   ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>>   AFTER MATCH SKIP PAST LAST ROW
>>   INITIAL
>>   PATTERN (LOWPRICE UP+ DOWN+)
>>   DEFINE
>>    LOWPRICE AS price &lt;= 100,
>>    UP AS price &gt; PREV(price),
>>    DOWN AS price &lt; PREV(price)
>> );
>> </programlisting>
>> ```
>>
>> PARTITION is at the same level as DEFINE, but it’s not defined as a
>> reserved keyword:
>> ```
>> PG_KEYWORD("partition", PARTITION, UNRESERVED_KEYWORD, BARE_LABEL)
>> ```
>>
>> Even in this patch,”initial”,”past”, “pattern” and “seek” are
>> defined as unreserved, why?
>>
>> So I just want to clarify.
> 
> 
> Because of position. Without making DEFINE a reserved keyword, how do
> you know that it isn't another variable in the PATTERN clause?

I think we don't need to worry about this because PATTERN_P is in the
$nonassoc list in the patch, which gives PATTERN different precedence
from DEFINE.

@@ -888,6 +896,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
 %nonassoc	UNBOUNDED NESTED /* ideally would have same precedence as IDENT */
 %nonassoc	IDENT PARTITION RANGE ROWS GROUPS PRECEDING FOLLOWING CUBE ROLLUP
 			SET KEYS OBJECT_P SCALAR VALUE_P WITH WITHOUT PATH
+			AFTER INITIAL SEEK PATTERN_P

And I think we could change DEFINE to an unreserved keyword.  Attached
is a patch to do that, on top of v35-0001.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  111. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2025-11-19T07:19:03Z

    > 2 - 0001 - gram.y
> ```
> opt_row_pattern_initial_or_seek:
> 			INITIAL		{ $$ = true; }
> 			| SEEK
> 				{
> 					ereport(ERROR,
> 							(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
> 							 errmsg("SEEK is not supported"),
> 							 errhint("Use INITIAL instead."),
> 							 parser_errposition(@1)));
> 				}
> 			| /*EMPTY*/		{ $$ = true; }
> 		;
> ```
> 
> As SEEK is specially listed here, I guess it might be supported in future. If that is true, would it be better to defer the semantic check to later parse phase, which would future work easier.

From a comment in gram.y:

/*
 * If we see PARTITION, RANGE, ROWS, GROUPS, AFTER, INITIAL, SEEK or PATTERN_P
 * as the first token after the '(' of a window_specification, we want the
 * assumption to be that there is no existing_window_name; but those keywords
 * are unreserved and so could be ColIds.  We fix this by making them have the

For this purpose, we want INITIAL and SEEK to be unreserved keywords.

> 3 - 0001 - parsenodes.h
> ```
> +/*
> + * RowPatternCommonSyntax - raw representation of row pattern common syntax
> + *
> + */
> +typedef struct RPCommonSyntax
> +{
> +	NodeTag		type;
> +	RPSkipTo	rpSkipTo;		/* Row Pattern AFTER MATCH SKIP type */
> +	bool		initial;		/* true if <row pattern initial or seek> is
> +								 * initial */
> +	List	   *rpPatterns;		/* PATTERN variables (list of A_Expr) */
> +	List	   *rpDefs;			/* row pattern definitions clause (list of
> +								 * ResTarget) */
> +} RPCommonSyntax;
> +
>  /*
>   * WindowDef - raw representation of WINDOW and OVER clauses
>   *
> @@ -593,6 +618,7 @@ typedef struct WindowDef
>  	char	   *refname;		/* referenced window name, if any */
>  	List	   *partitionClause;	/* PARTITION BY expression list */
>  	List	   *orderClause;	/* ORDER BY (list of SortBy) */
> +	RPCommonSyntax *rpCommonSyntax; /* row pattern common syntax */
> ```
> 
> RP fields are directly defined in WindowClause, then why do we need a wrapper of RPCommonSyntax? Can we directly define RP fields in WindowRef as well?

The row pattern common syntax defined in the standard is not only used
in the WINDOW clause, but in the FROM clause. If we would support RPR
in FROM clause in the future, it would be better to use the same code
of row pattern common syntax in WINDOW clause as much as
possible. That's the reason I created RPCommonSyntax. In the
parse/analysis phase, I am not sure how the parse/analysis code would
be in FROM clause at this point. So I did not define yet another
struct for it.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  112. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2025-11-19T07:20:46Z

    > 4 - 0001 - parsenodes.h
> ```
> +	/* Row Pattern AFTER MACH SKIP clause */
> ```
> 
> Typo: MACH -> MATCH

Thanks for pointing it out. Will fix.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  113. Re: Row pattern recognition

    Chao Li <li.evan.chao@gmail.com> — 2025-11-20T07:33:48Z

    
> On Nov 19, 2025, at 12:14, Chao Li <li.evan.chao@gmail.com> wrote:
> 
> 
> 9 - 0002 - parse_clause.c

I am continuing to review 0003

10 - 0003
```
+	Assert(list_length(patternVariables) == list_length(defineClause));
```

Is this assert true? From what I learned, pattern length doesn’t have to equal to define length. For example, I got an example from [1]:

```
Example 4
-- This example has three different patterns.
-- Comment two of them, to get error-free query.
SELECT company, price_date, price
  FROM stock_price
    MATCH_RECOGNIZE (
       partition by company
       order by price_date
       all rows per match
       pattern ( limit_50  up   up* down  down*  )
       define
           limit_50 as price <= 50.00,
           up   as price > prev(price),
           down as price < prev(price)
    )
   WHERE company = 'B'
   ORDER BY price_date;
```

In this example, pattern has 5 elements and define has only 3 elements.

11 - 0004 - plannodes.h
```
+	/* Row Pattern Recognition AFTER MACH SKIP clause */
+	RPSkipTo	rpSkipTo;		/* Row Pattern Skip To type */
```

Typo: MACH -> MATCH

I’d stop here, and continue 0005 tomorrow.


[1] https://link.springer.com/article/10.1007/s13222-022-00404-3

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

  114. Re: Row pattern recognition

    Chao Li <li.evan.chao@gmail.com> — 2025-11-21T05:25:50Z

    
> On Nov 20, 2025, at 15:33, Chao Li <li.evan.chao@gmail.com> wrote:
> 
> 
> I’d stop here, and continue 0005 tomorrow.
> 

I reviewed 0005 today. I'm still not very familiar with the executor code, so going through 0005 has also been a valuable learning process for me.

12 - 0005 - nodeWindowAgg.c
```
+	if (string_set_get_size(str_set) == 0)
+	{
+		/* no match found in the first row */
+		register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+		destroyStringInfo(encoded_str);
+		return;
+	}
```

encoded_str should also be destroyed if not entering the “if” clause.

13 - 0005 - nodeWindowAgg.c
```
static
int
do_pattern_match(char *pattern, char *encoded_str, int len)
{
static regex_t *regcache = NULL;
static regex_t preg;
static char patbuf[1024]; /* most recent 'pattern' is cached here */
```

Using static variable in executor seems something I have never seen, which may persistent data across queries. I think usually per query data are stored in state structures.

14 - 0005 - nodeWindowAgg.c
```
+		MemoryContext oldContext = MemoryContextSwitchTo(TopMemoryContext);
+
+		if (regcache != NULL)
+			pg_regfree(regcache);	/* free previous re */
+
+		/* we need to convert to char to pg_wchar */
+		plen = strlen(pattern);
+		data = (pg_wchar *) palloc((plen + 1) * sizeof(pg_wchar));
+		data_len = pg_mb2wchar_with_len(pattern, data, plen);
+		/* compile re */
+		sts = pg_regcomp(&preg, /* compiled re */
+						 data,	/* target pattern */
+						 data_len,	/* length of pattern */
+						 cflags,	/* compile option */
+						 C_COLLATION_OID	/* collation */
+			);
+		pfree(data);
+
+		MemoryContextSwitchTo(oldContext);
```

Here in do_pattern_match, it switches to top memory context and compiles the re and stores to “preg". Based on the comment of pg_regcomp:
```
/*
* pg_regcomp - compile regular expression
*
* Note: on failure, no resources remain allocated, so pg_regfree()
* need not be applied to re.
*/
int
pg_regcomp(regex_t *re,
const chr *string,
size_t len,
int flags,
Oid collation)
```

“preg” should be freed by pg_regfree(), given the memory is allocated in TopMemoryContext, this looks like a memory leak.

Okay, I’d stop here and continue to review 0006 next week.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

  115. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2025-11-21T05:57:08Z

    Hi Chao,

>> On Nov 18, 2025, at 19:19, Vik Fearing <vik@postgresfriends.org> wrote:
>> 
>> 
>> Because of position. Without making DEFINE a reserved keyword, how do you know that it isn't another variable in the PATTERN clause?
>> 
> 
> Ah, thanks for the clarification. Now I got it.
> 
> I’m continue to review 0002.

Thanks for the review!

> 5 - 0002 - parse_clause.c
> ```
> +		ereport(ERROR,
> +				(errcode(ERRCODE_SYNTAX_ERROR),
> +				 errmsg("FRAME must start at current row when row patttern recognition is used")));
> ```
> 
> Nit typo: patttern -> pattern

Will fix (this involves regression test change because this changes
the ERROR messages in the expected file).

> 6 - 0002 - parse_clause.c
> ```
> +	/* DEFINE variable name initials */
> +	static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz”;
> ```
> 
> This string can also be const, so “static const char *”

Agreed. Will fix.

> 7 - 0002 - parse_clause.c
> ```
> +	/*
> +	 * Create list of row pattern DEFINE variable name's initial. We assign
> +	 * [a-z] to them (up to 26 variable names are allowed).
> +	 */
> +	restargets = NIL;
> +	i = 0;
> +	initialLen = strlen(defineVariableInitials);
> +
> +	foreach(lc, windef->rpCommonSyntax->rpDefs)
> +	{
> +		char		initial[2];
> +
> +		restarget = (ResTarget *) lfirst(lc);
> +		name = restarget->name;
> ```
> 
> Looks like “name” is not used inside “foreach”.

Oops. Will fix.

> 8 - 0002 - parse_clause.c
> ```
> +	/*
> +	 * Create list of row pattern DEFINE variable name's initial. We assign
> +	 * [a-z] to them (up to 26 variable names are allowed).
> +	 */
> +	restargets = NIL;
> +	i = 0;
> +	initialLen = strlen(defineVariableInitials);
> +
> +	foreach(lc, windef->rpCommonSyntax->rpDefs)
> +	{
> +		char		initial[2];
> ```
> 
> I guess this “foreach” for build initial list can be combined into the previous foreach loop of checking duplication. If an def has no dup, then assign an initial to it.

You are right. Will change.

> 9 - 0002 - parse_clause.c
> ```
> +static void
> +transformPatternClause(ParseState *pstate, WindowClause *wc,
> +					   WindowDef *windef)
> +{
> +	ListCell   *lc;
> +
> +	/*
> +	 * Row Pattern Common Syntax clause exists?
> +	 */
> +	if (windef->rpCommonSyntax == NULL)
> +		return;
> ```
> 
> Checking  (windef->rpCommonSyntax == NULL) seems a duplicate, because transformPatternClause() is only called by transformRPR(), and transformRPR() has done the check.

Yeah. transformDefineClause() already does the similar check using
Assert. What about using Assert in transPatternClause() as well?

	Assert(windef->rpCommonSyntax != NULL);

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  116. Re: Row pattern recognition

    Chao Li <li.evan.chao@gmail.com> — 2025-11-24T00:59:57Z

    
> On Nov 21, 2025, at 13:25, Chao Li <li.evan.chao@gmail.com> wrote:
> 
> 
> Okay, I’d stop here and continue to review 0006 next week.
> 

I just finished reviewing 0006, and see some problems:

15 - 0006 - select.sgml
```
+[ <replaceable class="parameter">row_pattern_common_syntax</replaceable> ]
```

row_pattern_common_syntax doesn’t look like a good name. I searched over the docs, and couldn't find a name suffixed by “_syntax”. I think we can just name it as “row_pattern_recognition_clause” or a shorter name just “row_pattern”.

16 - 0006 - select.sgml
```
+ <synopsis>
+ [ AFTER MATCH SKIP PAST LAST ROW | AFTER MATCH SKIP TO NEXT ROW ]
``

I think the two values are mutually exclusive, so curly braces should added surround them:

[ { AFTER MATCH SKIP PAST LAST ROW | AFTER MATCH SKIP TO NEXT ROW } ]

[] means optional, and {} means choose one from all alternatives.

17 - 0006 - select.sgml
```
PATTERN <replaceable class="parameter">pattern_variable_name</replaceable>[+] [, ...]
```

PATTERN definition should be placed inside (), here you missed ()

18 - 0006 - select.sgml
The same code as 17, <replaceable class="parameter">pattern_variable_name</replaceable>[+], do you only put “+” here, I think SQL allows a lot of pattern quantifiers. From 0001, gram.y, looks like +, * and ? Are supported in this patch. So, maybe we can do:

```
PATTERN ( pattern_element, [pattern_element …] )

and pattern_element = variable name optionally followed by quantifier (reference to somewhere introducing pattern quantifier).
```

19 - 0006 - select.sgml

I don’t see INITIAL and SEEK are described. Even if SEEK is not supported currently, it’s worthy mentioning that.

20 - 0006 - select.sgml
```
+    after a match found. With <literal>AFTER MATCH SKIP PAST LAST
+    ROW</literal> (the default) next row position is next to the last row of
```

21 - 0006 - select.sgml
```
 [ <replaceable class="parameter">frame_clause</replaceable> ]
+[ <replaceable class="parameter">row_pattern_common_syntax</replaceable> ]
```

Looks like row_pattern_common_syntax belongs to frame_clause, and you have lately added row_pattern_common_syntax to frame_clause:
```
 <synopsis>
-{ RANGE | ROWS | GROUPS } <replaceable>frame_start</replaceable> [ <replaceable>frame_exclusion</replaceable> ]
-{ RANGE | ROWS | GROUPS } BETWEEN <replaceable>frame_start</replaceable> AND <replaceable>frame_end</replaceable> [ <replaceable>frame_exclusion</replaceable> ]
+{ RANGE | ROWS | GROUPS } <replaceable>frame_start</replaceable> [ <replaceable>frame_exclusion</replaceable> ] [row_pattern_common_syntax]
+{ RANGE | ROWS | GROUPS } BETWEEN <replaceable>frame_start</replaceable> AND <replaceable>frame_end</replaceable> [ <replaceable>frame_exclusion</replaceable> ] [row_pattern_common_syntax]
 </synopsis>
```

So I guess row_pattern_common_syntax after frame_clause should not be added.

22 - 0006 - advance.sgml
```
<programlisting>
DEFINE
 LOWPRICE AS price &lt;= 100,
 UP AS price &gt; PREV(price),
 DOWN AS price &lt; PREV(price)
</programlisting>

    Note that <function>PREV</function> returns the price column in the
```

In the “Note” line, “price” refers to a column from above example, so I think it should be tagged by <literal>. This comment applies to multiple places.

I will proceed 0007 tomorrow.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

  117. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2025-11-24T04:57:25Z

    Hi Chao,

Thank you for the review!

>> On Nov 20, 2025, at 15:33, Chao Li <li.evan.chao@gmail.com> wrote:
>> 
>> 
>> I’d stop here, and continue 0005 tomorrow.
>> 
> 
> I reviewed 0005 today. I'm still not very familiar with the executor code, so going through 0005 has also been a valuable learning process for me.
> 
> 12 - 0005 - nodeWindowAgg.c
> ```
> +	if (string_set_get_size(str_set) == 0)
> +	{
> +		/* no match found in the first row */
> +		register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
> +		destroyStringInfo(encoded_str);
> +		return;
> +	}
> ```
> 
> encoded_str should also be destroyed if not entering the “if” clause.

Subsequent string_set_discard() will free encode_str since encoded_str
is part of str_set. So no need to call destroyStringInfo(encoded_str)
in not entering "if" clause.

	string_set_discard(str_set);

So I think this is Ok.

> 13 - 0005 - nodeWindowAgg.c
> ```
> static
> int
> do_pattern_match(char *pattern, char *encoded_str, int len)
> {
> static regex_t *regcache = NULL;
> static regex_t preg;
> static char patbuf[1024]; /* most recent 'pattern' is cached here */
> ```
> 
> Using static variable in executor seems something I have never seen, which may persistent data across queries. I think usually per query data are stored in state structures.

Yeah, such a usage maybe rare. But I hesitate to store the data
(regex_t) in WindowAggState because:

(1) it bloats WindowAggState which we want to avoid if possible
(2) it requires execnodes.h to include regex.h. (maybe this itself is harmless, I am not sure)

> 14 - 0005 - nodeWindowAgg.c
> ```
> +		MemoryContext oldContext = MemoryContextSwitchTo(TopMemoryContext);
> +
> +		if (regcache != NULL)
> +			pg_regfree(regcache);	/* free previous re */
> +
> +		/* we need to convert to char to pg_wchar */
> +		plen = strlen(pattern);
> +		data = (pg_wchar *) palloc((plen + 1) * sizeof(pg_wchar));
> +		data_len = pg_mb2wchar_with_len(pattern, data, plen);
> +		/* compile re */
> +		sts = pg_regcomp(&preg, /* compiled re */
> +						 data,	/* target pattern */
> +						 data_len,	/* length of pattern */
> +						 cflags,	/* compile option */
> +						 C_COLLATION_OID	/* collation */
> +			);
> +		pfree(data);
> +
> +		MemoryContextSwitchTo(oldContext);
> ```
> 
> Here in do_pattern_match, it switches to top memory context and compiles the re and stores to “preg". Based on the comment of pg_regcomp:
> ```
> /*
> * pg_regcomp - compile regular expression
> *
> * Note: on failure, no resources remain allocated, so pg_regfree()
> * need not be applied to re.
> */
> int
> pg_regcomp(regex_t *re,
> const chr *string,
> size_t len,
> int flags,
> Oid collation)
> ```
> 
> “preg” should be freed by pg_regfree(), given the memory is allocated in TopMemoryContext, this looks like a memory leak.

I admit current patch leaves the memory unfreed even after a query
finishes. What about adding something like:

static void do_pattern_match_end(void)
{
	if (regcache != NULL)
		pg_regfree(regcache);
}

and let ExecEndWindowAgg() call it?

> Okay, I’d stop here and continue to review 0006 next week.
> 
> Best regards,
> --
> Chao Li (Evan)
> HighGo Software Co., Ltd.
> https://www.highgo.com/
> 
> 
> 
>

  118. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2025-11-24T06:47:03Z

    Hi Chao,

>> On Nov 21, 2025, at 13:25, Chao Li <li.evan.chao@gmail.com> wrote:
>> 
>> 
>> Okay, I’d stop here and continue to review 0006 next week.
>> 
> 
> I just finished reviewing 0006, and see some problems:
> 
> 15 - 0006 - select.sgml
> ```
> +[ <replaceable class="parameter">row_pattern_common_syntax</replaceable> ]
> ```
> 
> row_pattern_common_syntax doesn’t look like a good name. I searched over the docs, and couldn't find a name suffixed by “_syntax”. I think we can just name it as “row_pattern_recognition_clause” or a shorter name just “row_pattern”.

I believe these names are based on the SQL standard. All syntax
components do not necessary be suffixed by "clause". For example
in select.sgml,

[ WITH [ RECURSIVE ] <replaceable class="parameter">with_query</replaceable> [, ...] ]
SELECT [ ALL | DISTINCT [ ON ( <replaceable class="parameter">expression</replaceable> [, ...] ) ] ]
    [ { * | <replaceable class="parameter">expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
    [ FROM <replaceable class="parameter">from_item</replaceable> [, ...] ]
    [ WHERE <replaceable class="parameter">condition</replaceable> ]
    [ GROUP BY { ALL | [ ALL | DISTINCT ] <replaceable class="parameter">grouping_element</replaceable> [, ...] } ]
    [ HAVING <replaceable class="parameter">condition</replaceable> ]
    [ WINDOW <replaceable class="parameter">window_name</replaceable> AS ( <replaceable class="parameter">window_definition</replaceable> ) [, ...] ]

"window_definition" comes from the standard and does not suffixed by
"clause". If you look into the window clause definition in the
standard, you will see:

<window clause> ::=
WINDOW <window definition list>
<window definition list> ::=
<window definition> [ { <comma> <window definition> }... ]

So the usage of terms in our document matches the standard. Likewise,
we see the definition of row pattern common syntax in the stabdard:

<window clause> ::=
WINDOW <window definition list>
<window definition list> ::=
<window definition> [ { <comma> <window definition> }... ]
<window definition> ::=
<new window name> AS <window specification>
<new window name> ::=
<window name>
<window specification> ::=
<left paren> <window specification details> <right paren>
<window specification details> ::=
[ <existing window name> ]
[ <window partition clause> ]
[ <window order clause> ]
[ <window frame clause> ]
:
:
<window frame clause> ::=
[ <row pattern measures> ]
<window frame units> <window frame extent>
[ <window frame exclusion> ]
[ <row pattern common syntax> ]

So I think "row pattern common syntax" is perfectly appropriate
name. If we change it to “row_pattern_recognition_clause”, it will
just bring confusion.  In the standard “row pattern recognition
clause” is defined RPR in FROM clause.

SELECT ... FROM table MATCH RECOGNIZE (....);

Here MATCH RECOGNIZE (....) is the “row pattern recognition clause”.

> 16 - 0006 - select.sgml
> ```
> + <synopsis>
> + [ AFTER MATCH SKIP PAST LAST ROW | AFTER MATCH SKIP TO NEXT ROW ]
> ``
> 
> I think the two values are mutually exclusive, so curly braces should added surround them:
> 
> [ { AFTER MATCH SKIP PAST LAST ROW | AFTER MATCH SKIP TO NEXT ROW } ]
> 
> [] means optional, and {} means choose one from all alternatives.

Agreed. Will fix.

> 17 - 0006 - select.sgml
> ```
> PATTERN <replaceable class="parameter">pattern_variable_name</replaceable>[+] [, ...]
> ```
> 
> PATTERN definition should be placed inside (), here you missed ()

Good catch. Will fix.

> 18 - 0006 - select.sgml
> The same code as 17, <replaceable class="parameter">pattern_variable_name</replaceable>[+], do you only put “+” here, I think SQL allows a lot of pattern quantifiers. From 0001, gram.y, looks like +, * and ? Are supported in this patch. So, maybe we can do:
> 
> ```
> PATTERN ( pattern_element, [pattern_element …] )
> 
> and pattern_element = variable name optionally followed by quantifier (reference to somewhere introducing pattern quantifier).
> ```

Currently only *, + and ? are supported and I don't think it's worth
to invent "pattern_element".  (Actually the standard defines much more
complex syntax for PATTERN. I think we can revisit this once the basic
support for quantifier *,+ and ? are brought in the core PostgreSQL
code.

> 19 - 0006 - select.sgml
> 
> I don’t see INITIAL and SEEK are described. Even if SEEK is not supported currently, it’s worthy mentioning that.

Agreed. Will fix.

> 20 - 0006 - select.sgml
> ```
> +    after a match found. With <literal>AFTER MATCH SKIP PAST LAST
> +    ROW</literal> (the default) next row position is next to the last row of
> ```
> 
> 21 - 0006 - select.sgml
> ```
>  [ <replaceable class="parameter">frame_clause</replaceable> ]
> +[ <replaceable class="parameter">row_pattern_common_syntax</replaceable> ]
> ```
> 
> Looks like row_pattern_common_syntax belongs to frame_clause, and you have lately added row_pattern_common_syntax to frame_clause:
> ```
>  <synopsis>
> -{ RANGE | ROWS | GROUPS } <replaceable>frame_start</replaceable> [ <replaceable>frame_exclusion</replaceable> ]
> -{ RANGE | ROWS | GROUPS } BETWEEN <replaceable>frame_start</replaceable> AND <replaceable>frame_end</replaceable> [ <replaceable>frame_exclusion</replaceable> ]
> +{ RANGE | ROWS | GROUPS } <replaceable>frame_start</replaceable> [ <replaceable>frame_exclusion</replaceable> ] [row_pattern_common_syntax]
> +{ RANGE | ROWS | GROUPS } BETWEEN <replaceable>frame_start</replaceable> AND <replaceable>frame_end</replaceable> [ <replaceable>frame_exclusion</replaceable> ] [row_pattern_common_syntax]
>  </synopsis>
> ```
> 
> So I guess row_pattern_common_syntax after frame_clause should not be added.

Yes, row_pattern_common_syntax belongs to frame_clause. Will fix.

> 22 - 0006 - advance.sgml
> ```
> <programlisting>
> DEFINE
>  LOWPRICE AS price &lt;= 100,
>  UP AS price &gt; PREV(price),
>  DOWN AS price &lt; PREV(price)
> </programlisting>
> 
>     Note that <function>PREV</function> returns the price column in the
> ```
> 
> In the “Note” line, “price” refers to a column from above example, so I think it should be tagged by <literal>. This comment applies to multiple places.

You are right. Will fix.

> I will proceed 0007 tomorrow.

Thanks!

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  119. Re: Row pattern recognition

    Chao Li <li.evan.chao@gmail.com> — 2025-11-25T06:16:10Z

    
> On Nov 24, 2025, at 08:59, Chao Li <li.evan.chao@gmail.com> wrote:
> 
> I will proceed 0007 tomorrow.

I just finished reviewing 0007 and 0008. This patch set really demonstrates the full lifecycle of adding a SQL feature, from changing the syntax in gram.y all the way down to the executor, including tests and docs. I learned a lot from it. Thanks!

23 - 0007

As you mentioned earlier, pattern regular expression support *, + and ?, but I don’t see ? is tested.

24 - 0007

I don’t see negative tests for unsupported quantifiers, like PATTERN (A+?).

25 - 0007
```
+-- basic test with none greedy pattern
```

Typo: “none greedy” -> non-greedy

No comment for 0008.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

  120. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2025-11-27T01:16:42Z

    > I just finished reviewing 0007 and 0008. This patch set really demonstrates the full lifecycle of adding a SQL feature, from changing the syntax in gram.y all the way down to the executor, including tests and docs. I learned a lot from it. Thanks!

You are welcome!

> 23 - 0007
> 
> As you mentioned earlier, pattern regular expression support *, + and ?, but I don’t see ? is tested.

Good catch. I will add the test case. In the mean time I found a bug
with gram.y part which handles '?' quantifier.  gram.y can be
successfully compiled but there's no way to put '?' quantifier in a
SQL statement.  We cannot write directly "ColId '?'" like other
quantifier '*' and '+' in the grammer because '?' is not a "self"
token.  So we let '?' matches Op first then check if it's '?'  or
not. 

			| ColId Op
				{
					if (strcmp("?", $2))
						ereport(ERROR,
								errcode(ERRCODE_SYNTAX_ERROR),
								errmsg("unsupported quantifier"),
								parser_errposition(@2));

					$$ = (Node *) makeSimpleA_Expr(AEXPR_OP, "?", (Node *)makeString($1), NULL, @1);
				}

Another bug was with executor (nodeWindowAgg.c). The code to check
greedy quantifiers missed the case '?'.

> 24 - 0007
> 
> I don’t see negative tests for unsupported quantifiers, like PATTERN (A+?).

I will add such test cases.

> 25 - 0007
> ```
> +-- basic test with none greedy pattern
> ```
> 
> Typo: “none greedy” -> non-greedy

Will fix.

> No comment for 0008.

Thanks again for the review. I will post v35 patch set soon.  Attached
is the diff from v34.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  121. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2025-11-27T03:10:08Z

    Hi Chao,

Any comment on this?

>> 13 - 0005 - nodeWindowAgg.c
>> ```
>> static
>> int
>> do_pattern_match(char *pattern, char *encoded_str, int len)
>> {
>> static regex_t *regcache = NULL;
>> static regex_t preg;
>> static char patbuf[1024]; /* most recent 'pattern' is cached here */
>> ```
>> 
>> Using static variable in executor seems something I have never seen, which may persistent data across queries. I think usually per query data are stored in state structures.
>
>Yeah, such a usage maybe rare. But I hesitate to store the data
>(regex_t) in WindowAggState because:
>
>(1) it bloats WindowAggState which we want to avoid if possible
>(2) it requires execnodes.h to include regex.h. (maybe this itself is harmless, I am not sure)
>
>> 14 - 0005 - nodeWindowAgg.c
>> ```
>> +		MemoryContext oldContext = MemoryContextSwitchTo(TopMemoryContext);
>> +
>> +		if (regcache != NULL)
>> +			pg_regfree(regcache);	/* free previous re */
>> +
>> +		/* we need to convert to char to pg_wchar */
>> +		plen = strlen(pattern);
>> +		data = (pg_wchar *) palloc((plen + 1) * sizeof(pg_wchar));
>> +		data_len = pg_mb2wchar_with_len(pattern, data, plen);
>> +		/* compile re */
>> +		sts = pg_regcomp(&preg, /* compiled re */
>> +						 data,	/* target pattern */
>> +						 data_len,	/* length of pattern */
>> +						 cflags,	/* compile option */
>> +						 C_COLLATION_OID	/* collation */
>> +			);
>> +		pfree(data);
>> +
>> +		MemoryContextSwitchTo(oldContext);
>> ```
>> 
>> Here in do_pattern_match, it switches to top memory context and compiles the re and stores to “preg". Based on the comment of pg_regcomp:
>> ```
>> /*
>> * pg_regcomp - compile regular expression
>> *
>> * Note: on failure, no resources remain allocated, so pg_regfree()
>> * need not be applied to re.
>> */
>> int
>> pg_regcomp(regex_t *re,
>> const chr *string,
>> size_t len,
>> int flags,
>> Oid collation)
>> ```
>> 
>> “preg” should be freed by pg_regfree(), given the memory is allocated in TopMemoryContext, this looks like a memory leak.
>
>I admit current patch leaves the memory unfreed even after a query
>finishes. What about adding something like:
>
>static void do_pattern_match_end(void)
>{
>	if (regcache != NULL)
>		pg_regfree(regcache);
>}
>
>and let ExecEndWindowAgg() call it?

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  122. Re: Row pattern recognition

    Chao Li <li.evan.chao@gmail.com> — 2025-11-27T06:56:02Z

    
> On Nov 27, 2025, at 11:10, Tatsuo Ishii <ishii@postgresql.org> wrote:
> 
> Hi Chao,
> 
> Any comment on this?
> 
>>> 13 - 0005 - nodeWindowAgg.c
>>> ```
>>> static
>>> int
>>> do_pattern_match(char *pattern, char *encoded_str, int len)
>>> {
>>> static regex_t *regcache = NULL;
>>> static regex_t preg;
>>> static char patbuf[1024]; /* most recent 'pattern' is cached here */
>>> ```
>>> 
>>> Using static variable in executor seems something I have never seen, which may persistent data across queries. I think usually per query data are stored in state structures.
>> 
>> Yeah, such a usage maybe rare. But I hesitate to store the data
>> (regex_t) in WindowAggState because:
>> 
>> (1) it bloats WindowAggState which we want to avoid if possible
>> (2) it requires execnodes.h to include regex.h. (maybe this itself is harmless, I am not sure)

With the static function-scope variables, those data persist across queries, which is error prone. I am afraid that even if I let this pass, other reviewers might have the same concern.

Maybe define a sub structure, say WindowAggCache, and put a pointer of WindowAggCache in WindowAggState, and only allocate memory when a query needs to.

>> 
>>> 14 - 0005 - nodeWindowAgg.c
>>> ```
>>> + MemoryContext oldContext = MemoryContextSwitchTo(TopMemoryContext);
>>> +
>>> + if (regcache != NULL)
>>> + pg_regfree(regcache); /* free previous re */
>>> +
>>> + /* we need to convert to char to pg_wchar */
>>> + plen = strlen(pattern);
>>> + data = (pg_wchar *) palloc((plen + 1) * sizeof(pg_wchar));
>>> + data_len = pg_mb2wchar_with_len(pattern, data, plen);
>>> + /* compile re */
>>> + sts = pg_regcomp(&preg, /* compiled re */
>>> + data, /* target pattern */
>>> + data_len, /* length of pattern */
>>> + cflags, /* compile option */
>>> + C_COLLATION_OID /* collation */
>>> + );
>>> + pfree(data);
>>> +
>>> + MemoryContextSwitchTo(oldContext);
>>> ```
>>> 
>>> Here in do_pattern_match, it switches to top memory context and compiles the re and stores to “preg". Based on the comment of pg_regcomp:
>>> ```
>>> /*
>>> * pg_regcomp - compile regular expression
>>> *
>>> * Note: on failure, no resources remain allocated, so pg_regfree()
>>> * need not be applied to re.
>>> */
>>> int
>>> pg_regcomp(regex_t *re,
>>> const chr *string,
>>> size_t len,
>>> int flags,
>>> Oid collation)
>>> ```
>>> 
>>> “preg” should be freed by pg_regfree(), given the memory is allocated in TopMemoryContext, this looks like a memory leak.
>> 
>> I admit current patch leaves the memory unfreed even after a query
>> finishes. What about adding something like:
>> 
>> static void do_pattern_match_end(void)
>> {
>> if (regcache != NULL)
>> pg_regfree(regcache);
>> }
>> 
>> and let ExecEndWindowAgg() call it?
> 

I’m not sure. But I think if we move the data into WindowAggState, then I guess we don’t have to use TopmemoryContext here.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

  123. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2025-12-01T06:57:02Z

    Hi Chao,

Sorry, I missed this email.

>> 9 - 0002 - parse_clause.c
> 
> I am continuing to review 0003
> 
> 10 - 0003
> ```
> +	Assert(list_length(patternVariables) == list_length(defineClause));
> ```
> 
> Is this assert true? From what I learned, pattern length doesn’t have to equal to define length. For example, I got an example from [1]:

You are right. If PATTERN clause uses the same pattern variable more
than once (which is perfectly valid), the assertion fails. I will
remove the assersion and fix the subsequent forboth loop.

> ```
> Example 4
> -- This example has three different patterns.
> -- Comment two of them, to get error-free query.
> SELECT company, price_date, price
>   FROM stock_price
>     MATCH_RECOGNIZE (
>        partition by company
>        order by price_date
>        all rows per match
>        pattern ( limit_50  up   up* down  down*  )
>        define
>            limit_50 as price <= 50.00,
>            up   as price > prev(price),
>            down as price < prev(price)
>     )
>    WHERE company = 'B'
>    ORDER BY price_date;
> ```
> 
> In this example, pattern has 5 elements and define has only 3 elements.

Yes.

> 11 - 0004 - plannodes.h
> ```
> +	/* Row Pattern Recognition AFTER MACH SKIP clause */
> +	RPSkipTo	rpSkipTo;		/* Row Pattern Skip To type */
> ```
> 
> Typo: MACH -> MATCH

Will fix/

> I’d stop here, and continue 0005 tomorrow.

Thanks!

> [1] https://link.springer.com/article/10.1007/s13222-022-00404-3
> 
> Best regards,
> --
> Chao Li (Evan)
> HighGo Software Co., Ltd.
> https://www.highgo.com/
> 
> 
> 
>

  124. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2025-12-01T12:21:38Z

    >> On Nov 27, 2025, at 11:10, Tatsuo Ishii <ishii@postgresql.org> wrote:
>> 
>> Hi Chao,
>> 
>> Any comment on this?
>> 
>>>> 13 - 0005 - nodeWindowAgg.c
>>>> ```
>>>> static
>>>> int
>>>> do_pattern_match(char *pattern, char *encoded_str, int len)
>>>> {
>>>> static regex_t *regcache = NULL;
>>>> static regex_t preg;
>>>> static char patbuf[1024]; /* most recent 'pattern' is cached here */
>>>> ```
>>>> 
>>>> Using static variable in executor seems something I have never seen, which may persistent data across queries. I think usually per query data are stored in state structures.
>>> 
>>> Yeah, such a usage maybe rare. But I hesitate to store the data
>>> (regex_t) in WindowAggState because:
>>> 
>>> (1) it bloats WindowAggState which we want to avoid if possible
>>> (2) it requires execnodes.h to include regex.h. (maybe this itself is harmless, I am not sure)
> 
> With the static function-scope variables, those data persist across queries, which is error prone. I am afraid that even if I let this pass, other reviewers might have the same concern.
> 
> Maybe define a sub structure, say WindowAggCache, and put a pointer of WindowAggCache in WindowAggState, and only allocate memory when a query needs to.
> 
>>> 
>>>> 14 - 0005 - nodeWindowAgg.c
>>>> ```
>>>> + MemoryContext oldContext = MemoryContextSwitchTo(TopMemoryContext);
>>>> +
>>>> + if (regcache != NULL)
>>>> + pg_regfree(regcache); /* free previous re */
>>>> +
>>>> + /* we need to convert to char to pg_wchar */
>>>> + plen = strlen(pattern);
>>>> + data = (pg_wchar *) palloc((plen + 1) * sizeof(pg_wchar));
>>>> + data_len = pg_mb2wchar_with_len(pattern, data, plen);
>>>> + /* compile re */
>>>> + sts = pg_regcomp(&preg, /* compiled re */
>>>> + data, /* target pattern */
>>>> + data_len, /* length of pattern */
>>>> + cflags, /* compile option */
>>>> + C_COLLATION_OID /* collation */
>>>> + );
>>>> + pfree(data);
>>>> +
>>>> + MemoryContextSwitchTo(oldContext);
>>>> ```
>>>> 
>>>> Here in do_pattern_match, it switches to top memory context and compiles the re and stores to “preg". Based on the comment of pg_regcomp:
>>>> ```
>>>> /*
>>>> * pg_regcomp - compile regular expression
>>>> *
>>>> * Note: on failure, no resources remain allocated, so pg_regfree()
>>>> * need not be applied to re.
>>>> */
>>>> int
>>>> pg_regcomp(regex_t *re,
>>>> const chr *string,
>>>> size_t len,
>>>> int flags,
>>>> Oid collation)
>>>> ```
>>>> 
>>>> “preg” should be freed by pg_regfree(), given the memory is allocated in TopMemoryContext, this looks like a memory leak.
>>> 
>>> I admit current patch leaves the memory unfreed even after a query
>>> finishes. What about adding something like:
>>> 
>>> static void do_pattern_match_end(void)
>>> {
>>> if (regcache != NULL)
>>> pg_regfree(regcache);
>>> }
>>> 
>>> and let ExecEndWindowAgg() call it?
>> 
> 
> I’m not sure. But I think if we move the data into WindowAggState, then I guess we don’t have to use TopmemoryContext here.

I decided to add new fields to WindowAggState:

	/* regular expression compiled result cache. Used for RPR. */
	char	   *patbuf;			/* pattern to be compiled */
	regex_t		preg;			/* compiled re pattern */

Then allocate the memory for them using
winstate->ss.ps.ps_ExprContext->ecxt_per_query_memory;

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  125. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2025-12-01T12:42:18Z

    >> I just finished reviewing 0007 and 0008. This patch set really demonstrates the full lifecycle of adding a SQL feature, from changing the syntax in gram.y all the way down to the executor, including tests and docs. I learned a lot from it. Thanks!
> 
> You are welcome!
> 
>> 23 - 0007
>> 
>> As you mentioned earlier, pattern regular expression support *, + and ?, but I don’t see ? is tested.
> 
> Good catch. I will add the test case. In the mean time I found a bug
> with gram.y part which handles '?' quantifier.  gram.y can be
> successfully compiled but there's no way to put '?' quantifier in a
> SQL statement.  We cannot write directly "ColId '?'" like other
> quantifier '*' and '+' in the grammer because '?' is not a "self"
> token.  So we let '?' matches Op first then check if it's '?'  or
> not. 
> 
> 			| ColId Op
> 				{
> 					if (strcmp("?", $2))
> 						ereport(ERROR,
> 								errcode(ERRCODE_SYNTAX_ERROR),
> 								errmsg("unsupported quantifier"),
> 								parser_errposition(@2));
> 
> 					$$ = (Node *) makeSimpleA_Expr(AEXPR_OP, "?", (Node *)makeString($1), NULL, @1);
> 				}
> 
> Another bug was with executor (nodeWindowAgg.c). The code to check
> greedy quantifiers missed the case '?'.
> 
>> 24 - 0007
>> 
>> I don’t see negative tests for unsupported quantifiers, like PATTERN (A+?).
> 
> I will add such test cases.
> 
>> 25 - 0007
>> ```
>> +-- basic test with none greedy pattern
>> ```
>> 
>> Typo: “none greedy” -> non-greedy
> 
> Will fix.
> 
>> No comment for 0008.
> 
> Thanks again for the review. I will post v35 patch set soon.  Attached
> is the diff from v34.

Attached are the v35 patches for Row pattern recognition.  Chao Li
reviewed v34 thoroughly. Thank you! v35 reflects the review
comments. Major differences from v34 include:

- Make "DEFINE" an unreserved keyword. Previously it was a reserved keyword.
- Refactor transformDefineClause() to make two foreach loops into single foreach loop.
- Make '?' quantifier in PATTERN work as advertised. Test for '?' quantifier is added too.
- Unsupported quantifier test added.
- Fix get_rule_define().
- Fix memory leak related to regcomp.
- Move regcomp compiled result cache from static data to WindowAggState.
- Fix several typos.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  126. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2025-12-23T13:26:54Z

    Attached are the v36 patches, just for rebasing.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  127. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2025-12-31T12:10:53Z

    SQL/RPR Patch Review Report
============================

Patch: Row Pattern Recognition (SQL:2016)
Commitfest: https://commitfest.postgresql.org/patch/4460

Review Methodology:
  This review focused on quality assessment, not line-by-line code audit.
  Key code paths and quality issues were examined with surrounding context
  when concerns arose. Documentation files were reviewed with AI-assisted
  grammar and typo checking. Code coverage was measured using gcov and
  custom analysis tools.

Limitations:
  - Not a security audit or formal verification
  - Parser and executor internals reviewed at module level, not exhaustively
  - Performance characteristics not benchmarked


TABLE OF CONTENTS
-----------------

1. Executive Summary
2. Issues Found
   2.1 Critical / Major
   2.2 Minor Issues (Review Needed)
   2.3 Minor Issues (Style)
   2.4 Suggestions for Discussion
3. Test Coverage
   3.1 Covered Areas
   3.2 Untested Items
   3.3 Unimplemented Features (No Test Needed)
4. Code Coverage Analysis
   4.1 Overall Coverage
   4.2 Coverage by File
   4.3 Uncovered Code Risk Assessment
5. Commit Analysis
6. Recommendations
7. Conclusion


1. EXECUTIVE SUMMARY
--------------------

Overall assessment: GOOD

The SQL/RPR patch demonstrates solid implementation quality within its
defined scope. Code follows PostgreSQL coding standards (with minor style
issues), test coverage is comprehensive at 96.4%, and documentation is
thorough with only minor typos.

Rating by Area:
- Code Quality:     Good (PostgreSQL style compliant, 26 static style fixes
recommended)
- Test Coverage:    Good (96.4% line coverage, 28 test cases)
- Documentation:    Good (Complete, 1 minor typo)
- Build/Regress:    Pass (make check-world, 753 tests passed)


2. ISSUES FOUND
---------------

2.1 Critical / Major

#1 [Code] Greedy pattern combinatorial explosion
   Pattern: PATTERN (A+ B+ C+) with DEFINE A AS TRUE, B AS TRUE, C AS TRUE
   Impact: Memory exhaustion or infinite wait
   Recommendation: Add work_mem-based memory limit (error on exceed)

   Evidence - No memory limit in current code:
   - nodeWindowAgg.c:5718-5722 string_set_init(): palloc() unconditional
   - nodeWindowAgg.c:5741-5750 string_set_add(): set_size *= 2; repalloc()
unlimited
   - nodeWindowAgg.c:5095-5174 generate_patterns(): Triple loop, no limit

   Only work_mem usage in RPR (nodeWindowAgg.c:1297):
     winstate->buffer = tuplestore_begin_heap(false, false, work_mem);
   -> For tuple buffer, unrelated to pattern combination memory (StringSet)

2.2 Minor Issues (Review Needed)

#1 [Code] nodeWindowAgg.c:5849,5909,5912
   pos > NUM_ALPHABETS check - intent unclear, causes error at 28 PATTERN
elements

   Reproduction:
   - PATTERN (A B C ... Z A) (27 elements) -> OK
   - PATTERN (A B C ... Z A B) (28 elements) -> ERROR "initial is not valid
char: b"

2.3 Minor Issues (Style)

#1 [Code] nodeWindowAgg.c (25 blocks)
   #ifdef RPR_DEBUG -> recommend elog(DEBUG2, ...) or remove

#2 [Code] src/backend/executor/nodeWindowAgg.c
   static keyword on separate line (26 functions)

#3 [Code] src/backend/utils/adt/ruleutils.c
   Whitespace typo: "regexp !=NULL" -> "regexp != NULL"

#4 [Code] nodeWindowAgg.c:4619
   Error message case: "Unrecognized" -> "unrecognized" (PostgreSQL style)

#5 [Doc] doc/src/sgml/ref/select.sgml:1128
   Typo: "maximu" -> "maximum"

2.4 Suggestions for Discussion

#1 Incremental matching with streaming NFA redesign
   Benefits:
   - O(n) time complexity per row (current: exponential in worst case)
   - Bounded memory via state merging and context absorption
   - Natural extension for OR patterns, {n,m} quantifiers, nested groups
   - Enables MEASURES clause with incremental aggregates
   - Proven approach in CEP engines (Flink, Esper)


3. TEST COVERAGE
----------------

3.1 Covered Areas

- PATTERN clause: +, *, ? quantifiers (line 41, 71, 86)
- DEFINE clause: Variable conditions, auto-TRUE for missing (line 120-133)
- PREV/NEXT functions: Single argument (line 44, 173)
- AFTER MATCH SKIP: TO NEXT ROW (line 182), PAST LAST ROW (line 198)
- Aggregate integration: sum, avg, count, max, min (line 258-277)
- Window function integration: first_value, last_value, nth_value (line
34-35)
- JOIN/CTE: JOIN (line 313), WITH (line 324)
- View: VIEW creation, pg_get_viewdef (line 390-406)
- Error cases: 7 error scenarios (line 409-532)
- Large partition: 5000 row smoke test (line 360-387)
- ROWS BETWEEN offset: CURRENT ROW AND 2 FOLLOWING (line 244)

3.2 Untested Items

Feature                  Priority   Recommendation
-------------------------------------------------------------------------------
26 variable limit        Medium     Test 26 variables success, 27th
variable error
NULL value handling      Low        Test PREV(col) where col or previous
row is NULL

Sample test for 26 variable limit:

    -- Should fail with 27th variable (parser error, no table needed)
    SELECT * FROM (SELECT 1 AS x) t
    WINDOW w AS (
      ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
      PATTERN (a b c d e f g h i j k l m n o p q r s t u v w x y z aa)
      DEFINE a AS TRUE, b AS TRUE, c AS TRUE, d AS TRUE, e AS TRUE, f AS
TRUE,
             g AS TRUE, h AS TRUE, i AS TRUE, j AS TRUE, k AS TRUE, l AS
TRUE,
             m AS TRUE, n AS TRUE, o AS TRUE, p AS TRUE, q AS TRUE, r AS
TRUE,
             s AS TRUE, t AS TRUE, u AS TRUE, v AS TRUE, w AS TRUE, x AS
TRUE,
             y AS TRUE, z AS TRUE, aa AS TRUE
    );
    -- ERROR: number of row pattern definition variable names exceeds 26

Sample test for NULL handling:

    CREATE TEMP TABLE stock_null (company TEXT, tdate DATE, price INTEGER);
    INSERT INTO stock_null VALUES ('c1', '2023-07-01', 100);
    INSERT INTO stock_null VALUES ('c1', '2023-07-02', NULL);  -- NULL in
middle
    INSERT INTO stock_null VALUES ('c1', '2023-07-03', 200);
    INSERT INTO stock_null VALUES ('c1', '2023-07-04', 150);

    SELECT company, tdate, price, count(*) OVER w AS match_count
    FROM stock_null
    WINDOW w AS (
      PARTITION BY company
      ORDER BY tdate
      ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
      PATTERN (START UP DOWN)
      DEFINE START AS TRUE, UP AS price > PREV(price), DOWN AS price <
PREV(price)
    );
    -- Result: all rows show match_count = 0 (NULL breaks pattern matching)

3.3 Unimplemented Features (No Test Needed)

Per documentation (select.sgml:1133-1136):
- MEASURES clause: Not implemented
- SUBSET clause: Not implemented
- AFTER MATCH variants: Only SKIP TO NEXT ROW, PAST LAST ROW supported

Not in documentation, verified by testing:
- {n,m} quantifier: Parser error ("syntax error at or near {")
- (A|B) OR pattern: Not supported (parsed as single variable name "a|b")
- (A B)+ compound repetition: Parser accepts, but execution returns 0
matches


4. CODE COVERAGE ANALYSIS
-------------------------

4.1 Overall Coverage

96.4% (732 / 759 lines)

4.2 Coverage by File (major RPR-modified files)

nodeWindowAgg.c:    96.6% (560/580) - Pattern matching execution engine
parse_clause.c:     96.7% (88/91) - PATTERN/DEFINE analysis
ruleutils.c:        93.1% (54/58) - pg_get_viewdef output

4.3 Uncovered Code Risk Assessment

Total: 27 lines uncovered

Medium Risk (2 items) - Test addition recommended (see section 3.2):
- parse_clause.c:4093: transformDefineClause - 27th variable error
- nodeWindowAgg.c:5623: get_slots - null_slot for PREV at partition first
row

Low Risk (25 items) - Defensive code / Unreachable:
- nodeWindowAgg.c:3074: attno_map_walker - Parser validates arg count
- nodeWindowAgg.c:3137-3138: rpr_navigation_walker - switch default
- nodeWindowAgg.c:3566: window_gettupleslot - Before mark position
- nodeWindowAgg.c:4289: WinGetFuncArgInFrame - isout flag
- nodeWindowAgg.c:4393: WinGetSlotInFrame - Boundary check
- nodeWindowAgg.c:4618-4619: row_is_in_reduced_frame - switch default
- nodeWindowAgg.c:4697: register_reduced_frame_map - pos < 0 check
- nodeWindowAgg.c:5007: search_str_set - NULL return continue
- nodeWindowAgg.c:5405: do_pattern_match - Regex compile error
- nodeWindowAgg.c:5435,5437-5438: do_pattern_match - Regex exec error
- nodeWindowAgg.c:5700: pattern_initial - Variable not found
- nodeWindowAgg.c:5776: string_set_get - Index range check
- nodeWindowAgg.c:5850: variable_pos_register - a-z range check
- nodeWindowAgg.c:5910: variable_pos_fetch - a-z range check
- nodeWindowAgg.c:5913: variable_pos_fetch - Index range check
- parse_clause.c:3989: transformDefineClause - A_Expr type check
- parse_clause.c:4145: transformPatternClause - A_Expr type check
- ruleutils.c:6904-6908: get_rule_windowspec - SKIP TO NEXT ROW output

Conclusion: Most uncovered code consists of defensive error handling or
code unreachable due to parser pre-validation. No security or functional
risk.


5. COMMIT ANALYSIS
------------------

8 sequential commits:

Commit  Area            Files   +/-         Key Content
-------------------------------------------------------------------------------
1       Raw Parser      4       +174/-16    gram.y grammar (PATTERN/DEFINE)
2       Parse/Analysis  4       +277/-1     parse_clause.c analysis logic
3       Rewriter        1       +109/-0     pg_get_viewdef extension
4       Planner         5       +73/-3      WindowAgg plan node extension
5       Executor        4       +1,942/-11  CORE: Pattern matching engine
(+1,850)
6       Docs            3       +192/-7     advanced.sgml, func-window.sgml
7       Tests           3       +1,585/-1   rpr.sql (532), rpr.out (1,052)
8       typedefs        1       +6/-0       pgindent support

Code Change Statistics:
- Total files: 25
- Lines added: 4,358
- Lines deleted: 39
- Net increase: +4,319 lines


6. RECOMMENDATIONS
------------------

6.1 Combinatorial Explosion Prevention (Major, Required)

Add work_mem-based memory limit for StringSet allocation.
Location: string_set_add() in nodeWindowAgg.c:5741-5750
Consistent with existing PostgreSQL approach (Hash Join, Sort, etc.)

6.2 Code Review Required (Minor, Decision Needed)

Location: nodeWindowAgg.c:5849,5909,5912
Issue: pos > NUM_ALPHABETS check intent unclear
Current: PATTERN with 28 elements causes error
Question: Is 27 element limit intentional?

6.3 Code Style Fixes (Minor)

- #ifdef RPR_DEBUG: Use elog(DEBUG2, ...) or remove (25 blocks)
- static keyword on separate line: 26 functions to fix
- Whitespace: "regexp !=NULL" -> "regexp != NULL"
- Error message case: "Unrecognized" -> "unrecognized"

6.4 Documentation Fixes (Minor)

- select.sgml: "maximu" -> "maximum"

6.5 Test Additions (Optional)

Black-box Tests (Functional):

Feature              Test Case                                    Priority
-------------------------------------------------------------------------------
Variable limit       26 variables success, 27 error               Medium
NULL boundary        PREV at partition first row                  Medium

White-box Tests (Coverage) - covered by above black-box tests:

File:Line                          Target Branch
-------------------------------------------------------------------------------
parse_clause.c:4093                Limit error branch (Variable limit test)
nodeWindowAgg.c:5623               null_slot usage (NULL boundary test)


7. CONCLUSION
-------------

Test Quality: GOOD

Core functionality is thoroughly tested with comprehensive error case
coverage.

The patch is well-implemented within its defined scope. Identified issues
include
one major concern (combinatorial explosion) and minor style/documentation
items.

Recommended actions before commit:
1. Add work_mem-based memory limit for pattern combinations (required)
2. Clarify pos > NUM_ALPHABETS check intent (review needed)
3. Fix code style issues (optional but recommended)
4. Fix documentation typo (trivial)
5. Add tests for variable limit and NULL handling (optional)

Points for discussion (optional):
6. Incremental matching with streaming NFA redesign


Attachment:
- coverage.tgz (gcov HTML report, RPR-modified code only)


---
End of Report

2025년 9월 24일 (수) PM 7:36, Tatsuo Ishii <ishii@postgresql.org>님이 작성:

> Attached are the v33 patches for Row pattern recognition.  The
> difference from v32 is that the raw parse tree printing patch is not
> included in v33. This is because now that we have
> debug_print_raw_parse.
>
> Best regards,
> --
> Tatsuo Ishii
> SRA OSS K.K.
> English: http://www.sraoss.co.jp/index_en/
> Japanese:http://www.sraoss.co.jp
>

  128. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-01T06:45:14Z

    Hi hackers,

I ran Valgrind memcheck against the RPR (Row Pattern Recognition)
implementation
to verify there are no memory issues introduced by the new Row Pattern
Recognition feature.

== Test Environment ==

- PostgreSQL: 19devel (master branch)
- Valgrind: 3.22.0
- Platform: Linux x86_64 (RHEL 9)
- Tests executed: rpr.sql

== Executive Summary ==

  +----------------------------------------------------------+
  | RPR Implementation: NO MEMORY ISSUES DETECTED            |
  +----------------------------------------------------------+

All detected leaks originate from existing PostgreSQL core components.
No RPR-related functions appear in any leak stack traces.

== Backend Process 541453: Itemized Leak Analysis ==

Total: 37 bytes definitely lost in 1 block, 1 unique leak context

+================================================================================+
| #  | Bytes | Blocks | Allocator | Root Cause       | RPR? | Verdict
    |
+================================================================================+
| 1  | 37    | 1      | strdup    | Postmaster init  | NO   | HARMLESS
     |
+================================================================================+
     TOTAL: 37 bytes in 1 block
     RPR-RELATED LEAKS: 0

== RPR Verification ==

All leak stack traces were examined. Key functions NOT present:

  - ExecInitWindowAgg (RPR path)
  - row_pattern_initial, row_pattern_final
  - Any function from src/backend/executor/nodeWindowAgg.c (RPR code paths)
  - Any RPR-related parser functions (PATTERN, DEFINE)

The RPR tests exercised window functions with PATTERN, DEFINE clauses
(e.g., PATTERN (START UP+ DOWN+)), yet none of these code paths appear in
leaks.

== Process Summary ==

+--------+----------------+-----------------+----------+--------------------+
| PID    | Type           | Definitely Lost | Errors   | Notes
 |
+--------+----------------+-----------------+----------+--------------------+
| 541303 | postmaster     | 37 bytes        | 1        | strdup at startup
 |
| 541453 | backend        | 37 bytes        | 12       | See above analysis
|
| 541314 | checkpointer   | 37 bytes        | 1        | strdup at startup
 |
| 541315 | bgwriter       | 37 bytes        | 1        | strdup at startup
 |
| 541316 | walwriter      | 37 bytes        | 1        | strdup at startup
 |
| 541317 | autovacuum     | 37 bytes        | 1        | strdup at startup
 |
| 541318 | logical rep    | 37 bytes        | 1        | strdup at startup
 |
| 541319 | syslogger      | 37 bytes        | 1        | strdup at startup
 |
| Others | aux workers    | 37 bytes each   | 1 each   | strdup at startup
 |
+--------+----------------+-----------------+----------+--------------------+

== Possibly Lost (LLVM JIT) ==

Additionally, ~16KB in ~165 blocks reported as "possibly lost" from LLVM
JIT:
  - llvm::MDTuple, llvm::User allocations
  - Origin: llvm_compile_expr -> jit_compile_expr -> ExecReadyExpr
  - These are LLVM's internal caches, not PostgreSQL leaks

== Other Memory Errors ==

+--------------------------------------------------------------------------+
| NO memory access errors detected:                                        |
|   - Invalid read/write: 0                                                |
|   - Use of uninitialized value: 0                                        |
|   - Conditional jump on uninitialized value: 0                           |
|   - Syscall param errors: 0                                              |
|   - Overlap in memcpy/strcpy: 0                                          |
+--------------------------------------------------------------------------+

Suppressed errors (via valgrind.supp):
  - 541453 (backend): 46 errors from 3 contexts suppressed
  - Other processes: ~0 errors each suppressed

Suppression rules in valgrind.supp (12 categories):
  1. Padding        - write() syscalls with uninitialized struct padding
  2. CRC            - CRC32 calculation on padded buffers
  3. Overlap        - memcpy overlap in bootstrap/relmap
  4. glibc          - Dynamic loader internals
  5. Regex/Pattern  - RE_compile_and_cache, patternsel
  6. Planner        - query_planner memcpy overlap
  7. Cache          - RelCache, CatCache, TypeCache possible leaks
  8. XLogReader     - WAL reader allocations
  9. Parallel       - ParallelWorkerMain allocations
  10. AutoVacuum    - AutoVacWorkerMain allocations
  11. GUC/Backend   - set_config_option, InitPostgres
  12. PL/pgSQL      - plpgsql_compile, SPI_prepare, CachedPlan

All suppressed errors are known, harmless PostgreSQL behaviors.
See attached valgrind.supp for details.

== Conclusion ==

The RPR implementation introduces NO memory leaks. All detected issues
are pre-existing PostgreSQL behaviors related to:

  1. Postmaster startup allocations
  2. LLVM JIT internal caching

These are by-design patterns that trade minimal memory for performance.

== Files Attached ==

- valgrind.supp: Suppression file used for this test
- 541303.log: Postmaster log
- 541453.log: Backend log with full leak details (56KB)
- 5413xx.log: Auxiliary process logs

--
Regards

2025년 12월 23일 (화) PM 10:28, Tatsuo Ishii <ishii@postgresql.org>님이 작성:

> Attached are the v36 patches, just for rebasing.
>
> Best regards,
> --
> Tatsuo Ishii
> SRA OSS K.K.
> English: http://www.sraoss.co.jp/index_en/
> Japanese:http://www.sraoss.co.jp
>

  129. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-02T01:26:49Z

    Hi Henson Choi,

> Hi hackers,
> 
> I ran Valgrind memcheck against the RPR (Row Pattern Recognition)
> implementation
> to verify there are no memory issues introduced by the new Row Pattern
> Recognition feature.

Thanks for the test!

> == Conclusion ==
> 
> The RPR implementation introduces NO memory leaks.

Glad to hear that.
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  130. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-02T02:16:15Z

    Hi Henson,

Thank you for the review! I will take care the issues (it will take
sometime).

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

> SQL/RPR Patch Review Report
> ============================
> 
> Patch: Row Pattern Recognition (SQL:2016)
> Commitfest: https://commitfest.postgresql.org/patch/4460
> 
> Review Methodology:
>   This review focused on quality assessment, not line-by-line code audit.
>   Key code paths and quality issues were examined with surrounding context
>   when concerns arose. Documentation files were reviewed with AI-assisted
>   grammar and typo checking. Code coverage was measured using gcov and
>   custom analysis tools.
> 
> Limitations:
>   - Not a security audit or formal verification
>   - Parser and executor internals reviewed at module level, not exhaustively
>   - Performance characteristics not benchmarked
> 
> 
> TABLE OF CONTENTS
> -----------------
> 
> 1. Executive Summary
> 2. Issues Found
>    2.1 Critical / Major
>    2.2 Minor Issues (Review Needed)
>    2.3 Minor Issues (Style)
>    2.4 Suggestions for Discussion
> 3. Test Coverage
>    3.1 Covered Areas
>    3.2 Untested Items
>    3.3 Unimplemented Features (No Test Needed)
> 4. Code Coverage Analysis
>    4.1 Overall Coverage
>    4.2 Coverage by File
>    4.3 Uncovered Code Risk Assessment
> 5. Commit Analysis
> 6. Recommendations
> 7. Conclusion
> 
> 
> 1. EXECUTIVE SUMMARY
> --------------------
> 
> Overall assessment: GOOD
> 
> The SQL/RPR patch demonstrates solid implementation quality within its
> defined scope. Code follows PostgreSQL coding standards (with minor style
> issues), test coverage is comprehensive at 96.4%, and documentation is
> thorough with only minor typos.
> 
> Rating by Area:
> - Code Quality:     Good (PostgreSQL style compliant, 26 static style fixes
> recommended)
> - Test Coverage:    Good (96.4% line coverage, 28 test cases)
> - Documentation:    Good (Complete, 1 minor typo)
> - Build/Regress:    Pass (make check-world, 753 tests passed)
> 
> 
> 2. ISSUES FOUND
> ---------------
> 
> 2.1 Critical / Major
> 
> #1 [Code] Greedy pattern combinatorial explosion
>    Pattern: PATTERN (A+ B+ C+) with DEFINE A AS TRUE, B AS TRUE, C AS TRUE
>    Impact: Memory exhaustion or infinite wait
>    Recommendation: Add work_mem-based memory limit (error on exceed)
> 
>    Evidence - No memory limit in current code:
>    - nodeWindowAgg.c:5718-5722 string_set_init(): palloc() unconditional
>    - nodeWindowAgg.c:5741-5750 string_set_add(): set_size *= 2; repalloc()
> unlimited
>    - nodeWindowAgg.c:5095-5174 generate_patterns(): Triple loop, no limit
> 
>    Only work_mem usage in RPR (nodeWindowAgg.c:1297):
>      winstate->buffer = tuplestore_begin_heap(false, false, work_mem);
>    -> For tuple buffer, unrelated to pattern combination memory (StringSet)
> 
> 2.2 Minor Issues (Review Needed)
> 
> #1 [Code] nodeWindowAgg.c:5849,5909,5912
>    pos > NUM_ALPHABETS check - intent unclear, causes error at 28 PATTERN
> elements
> 
>    Reproduction:
>    - PATTERN (A B C ... Z A) (27 elements) -> OK
>    - PATTERN (A B C ... Z A B) (28 elements) -> ERROR "initial is not valid
> char: b"
> 
> 2.3 Minor Issues (Style)
> 
> #1 [Code] nodeWindowAgg.c (25 blocks)
>    #ifdef RPR_DEBUG -> recommend elog(DEBUG2, ...) or remove
> 
> #2 [Code] src/backend/executor/nodeWindowAgg.c
>    static keyword on separate line (26 functions)
> 
> #3 [Code] src/backend/utils/adt/ruleutils.c
>    Whitespace typo: "regexp !=NULL" -> "regexp != NULL"
> 
> #4 [Code] nodeWindowAgg.c:4619
>    Error message case: "Unrecognized" -> "unrecognized" (PostgreSQL style)
> 
> #5 [Doc] doc/src/sgml/ref/select.sgml:1128
>    Typo: "maximu" -> "maximum"
> 
> 2.4 Suggestions for Discussion
> 
> #1 Incremental matching with streaming NFA redesign
>    Benefits:
>    - O(n) time complexity per row (current: exponential in worst case)
>    - Bounded memory via state merging and context absorption
>    - Natural extension for OR patterns, {n,m} quantifiers, nested groups
>    - Enables MEASURES clause with incremental aggregates
>    - Proven approach in CEP engines (Flink, Esper)
> 
> 
> 3. TEST COVERAGE
> ----------------
> 
> 3.1 Covered Areas
> 
> - PATTERN clause: +, *, ? quantifiers (line 41, 71, 86)
> - DEFINE clause: Variable conditions, auto-TRUE for missing (line 120-133)
> - PREV/NEXT functions: Single argument (line 44, 173)
> - AFTER MATCH SKIP: TO NEXT ROW (line 182), PAST LAST ROW (line 198)
> - Aggregate integration: sum, avg, count, max, min (line 258-277)
> - Window function integration: first_value, last_value, nth_value (line
> 34-35)
> - JOIN/CTE: JOIN (line 313), WITH (line 324)
> - View: VIEW creation, pg_get_viewdef (line 390-406)
> - Error cases: 7 error scenarios (line 409-532)
> - Large partition: 5000 row smoke test (line 360-387)
> - ROWS BETWEEN offset: CURRENT ROW AND 2 FOLLOWING (line 244)
> 
> 3.2 Untested Items
> 
> Feature                  Priority   Recommendation
> -------------------------------------------------------------------------------
> 26 variable limit        Medium     Test 26 variables success, 27th
> variable error
> NULL value handling      Low        Test PREV(col) where col or previous
> row is NULL
> 
> Sample test for 26 variable limit:
> 
>     -- Should fail with 27th variable (parser error, no table needed)
>     SELECT * FROM (SELECT 1 AS x) t
>     WINDOW w AS (
>       ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>       PATTERN (a b c d e f g h i j k l m n o p q r s t u v w x y z aa)
>       DEFINE a AS TRUE, b AS TRUE, c AS TRUE, d AS TRUE, e AS TRUE, f AS
> TRUE,
>              g AS TRUE, h AS TRUE, i AS TRUE, j AS TRUE, k AS TRUE, l AS
> TRUE,
>              m AS TRUE, n AS TRUE, o AS TRUE, p AS TRUE, q AS TRUE, r AS
> TRUE,
>              s AS TRUE, t AS TRUE, u AS TRUE, v AS TRUE, w AS TRUE, x AS
> TRUE,
>              y AS TRUE, z AS TRUE, aa AS TRUE
>     );
>     -- ERROR: number of row pattern definition variable names exceeds 26
> 
> Sample test for NULL handling:
> 
>     CREATE TEMP TABLE stock_null (company TEXT, tdate DATE, price INTEGER);
>     INSERT INTO stock_null VALUES ('c1', '2023-07-01', 100);
>     INSERT INTO stock_null VALUES ('c1', '2023-07-02', NULL);  -- NULL in
> middle
>     INSERT INTO stock_null VALUES ('c1', '2023-07-03', 200);
>     INSERT INTO stock_null VALUES ('c1', '2023-07-04', 150);
> 
>     SELECT company, tdate, price, count(*) OVER w AS match_count
>     FROM stock_null
>     WINDOW w AS (
>       PARTITION BY company
>       ORDER BY tdate
>       ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>       PATTERN (START UP DOWN)
>       DEFINE START AS TRUE, UP AS price > PREV(price), DOWN AS price <
> PREV(price)
>     );
>     -- Result: all rows show match_count = 0 (NULL breaks pattern matching)
> 
> 3.3 Unimplemented Features (No Test Needed)
> 
> Per documentation (select.sgml:1133-1136):
> - MEASURES clause: Not implemented
> - SUBSET clause: Not implemented
> - AFTER MATCH variants: Only SKIP TO NEXT ROW, PAST LAST ROW supported
> 
> Not in documentation, verified by testing:
> - {n,m} quantifier: Parser error ("syntax error at or near {")
> - (A|B) OR pattern: Not supported (parsed as single variable name "a|b")
> - (A B)+ compound repetition: Parser accepts, but execution returns 0
> matches
> 
> 
> 4. CODE COVERAGE ANALYSIS
> -------------------------
> 
> 4.1 Overall Coverage
> 
> 96.4% (732 / 759 lines)
> 
> 4.2 Coverage by File (major RPR-modified files)
> 
> nodeWindowAgg.c:    96.6% (560/580) - Pattern matching execution engine
> parse_clause.c:     96.7% (88/91) - PATTERN/DEFINE analysis
> ruleutils.c:        93.1% (54/58) - pg_get_viewdef output
> 
> 4.3 Uncovered Code Risk Assessment
> 
> Total: 27 lines uncovered
> 
> Medium Risk (2 items) - Test addition recommended (see section 3.2):
> - parse_clause.c:4093: transformDefineClause - 27th variable error
> - nodeWindowAgg.c:5623: get_slots - null_slot for PREV at partition first
> row
> 
> Low Risk (25 items) - Defensive code / Unreachable:
> - nodeWindowAgg.c:3074: attno_map_walker - Parser validates arg count
> - nodeWindowAgg.c:3137-3138: rpr_navigation_walker - switch default
> - nodeWindowAgg.c:3566: window_gettupleslot - Before mark position
> - nodeWindowAgg.c:4289: WinGetFuncArgInFrame - isout flag
> - nodeWindowAgg.c:4393: WinGetSlotInFrame - Boundary check
> - nodeWindowAgg.c:4618-4619: row_is_in_reduced_frame - switch default
> - nodeWindowAgg.c:4697: register_reduced_frame_map - pos < 0 check
> - nodeWindowAgg.c:5007: search_str_set - NULL return continue
> - nodeWindowAgg.c:5405: do_pattern_match - Regex compile error
> - nodeWindowAgg.c:5435,5437-5438: do_pattern_match - Regex exec error
> - nodeWindowAgg.c:5700: pattern_initial - Variable not found
> - nodeWindowAgg.c:5776: string_set_get - Index range check
> - nodeWindowAgg.c:5850: variable_pos_register - a-z range check
> - nodeWindowAgg.c:5910: variable_pos_fetch - a-z range check
> - nodeWindowAgg.c:5913: variable_pos_fetch - Index range check
> - parse_clause.c:3989: transformDefineClause - A_Expr type check
> - parse_clause.c:4145: transformPatternClause - A_Expr type check
> - ruleutils.c:6904-6908: get_rule_windowspec - SKIP TO NEXT ROW output
> 
> Conclusion: Most uncovered code consists of defensive error handling or
> code unreachable due to parser pre-validation. No security or functional
> risk.
> 
> 
> 5. COMMIT ANALYSIS
> ------------------
> 
> 8 sequential commits:
> 
> Commit  Area            Files   +/-         Key Content
> -------------------------------------------------------------------------------
> 1       Raw Parser      4       +174/-16    gram.y grammar (PATTERN/DEFINE)
> 2       Parse/Analysis  4       +277/-1     parse_clause.c analysis logic
> 3       Rewriter        1       +109/-0     pg_get_viewdef extension
> 4       Planner         5       +73/-3      WindowAgg plan node extension
> 5       Executor        4       +1,942/-11  CORE: Pattern matching engine
> (+1,850)
> 6       Docs            3       +192/-7     advanced.sgml, func-window.sgml
> 7       Tests           3       +1,585/-1   rpr.sql (532), rpr.out (1,052)
> 8       typedefs        1       +6/-0       pgindent support
> 
> Code Change Statistics:
> - Total files: 25
> - Lines added: 4,358
> - Lines deleted: 39
> - Net increase: +4,319 lines
> 
> 
> 6. RECOMMENDATIONS
> ------------------
> 
> 6.1 Combinatorial Explosion Prevention (Major, Required)
> 
> Add work_mem-based memory limit for StringSet allocation.
> Location: string_set_add() in nodeWindowAgg.c:5741-5750
> Consistent with existing PostgreSQL approach (Hash Join, Sort, etc.)
> 
> 6.2 Code Review Required (Minor, Decision Needed)
> 
> Location: nodeWindowAgg.c:5849,5909,5912
> Issue: pos > NUM_ALPHABETS check intent unclear
> Current: PATTERN with 28 elements causes error
> Question: Is 27 element limit intentional?
> 
> 6.3 Code Style Fixes (Minor)
> 
> - #ifdef RPR_DEBUG: Use elog(DEBUG2, ...) or remove (25 blocks)
> - static keyword on separate line: 26 functions to fix
> - Whitespace: "regexp !=NULL" -> "regexp != NULL"
> - Error message case: "Unrecognized" -> "unrecognized"
> 
> 6.4 Documentation Fixes (Minor)
> 
> - select.sgml: "maximu" -> "maximum"
> 
> 6.5 Test Additions (Optional)
> 
> Black-box Tests (Functional):
> 
> Feature              Test Case                                    Priority
> -------------------------------------------------------------------------------
> Variable limit       26 variables success, 27 error               Medium
> NULL boundary        PREV at partition first row                  Medium
> 
> White-box Tests (Coverage) - covered by above black-box tests:
> 
> File:Line                          Target Branch
> -------------------------------------------------------------------------------
> parse_clause.c:4093                Limit error branch (Variable limit test)
> nodeWindowAgg.c:5623               null_slot usage (NULL boundary test)
> 
> 
> 7. CONCLUSION
> -------------
> 
> Test Quality: GOOD
> 
> Core functionality is thoroughly tested with comprehensive error case
> coverage.
> 
> The patch is well-implemented within its defined scope. Identified issues
> include
> one major concern (combinatorial explosion) and minor style/documentation
> items.
> 
> Recommended actions before commit:
> 1. Add work_mem-based memory limit for pattern combinations (required)
> 2. Clarify pos > NUM_ALPHABETS check intent (review needed)
> 3. Fix code style issues (optional but recommended)
> 4. Fix documentation typo (trivial)
> 5. Add tests for variable limit and NULL handling (optional)
> 
> Points for discussion (optional):
> 6. Incremental matching with streaming NFA redesign
> 
> 
> Attachment:
> - coverage.tgz (gcov HTML report, RPR-modified code only)
> 
> 
> ---
> End of Report
> 
> 2025년 9월 24일 (수) PM 7:36, Tatsuo Ishii <ishii@postgresql.org>님이 작성:
> 
>> Attached are the v33 patches for Row pattern recognition.  The
>> difference from v32 is that the raw parse tree printing patch is not
>> included in v33. This is because now that we have
>> debug_print_raw_parse.
>>
>> Best regards,
>> --
>> Tatsuo Ishii
>> SRA OSS K.K.
>> English: http://www.sraoss.co.jp/index_en/
>> Japanese:http://www.sraoss.co.jp
>>

  131. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-02T02:30:31Z

    Hi Ishii-san,

Glad the review was useful!

2026년 1월 2일 (금) AM 11:16, Tatsuo Ishii <ishii@postgresql.org>님이 작성:

> Hi Henson,
>
> Thank you for the review! I will take care the issues (it will take
> sometime).
>
>
If you have other patches that need pre-analysis (coverage, style,
valgrind, etc.), feel free to ask anytime. I enjoy this kind of work.


> Best regards,
> --
> Tatsuo Ishii
> SRA OSS K.K.
> English: http://www.sraoss.co.jp/index_en/
> Japanese:http://www.sraoss.co.jp
>
> > SQL/RPR Patch Review Report
> > ============================
> >
> > Patch: Row Pattern Recognition (SQL:2016)
> > Commitfest: https://commitfest.postgresql.org/patch/4460
> >
> > Review Methodology:
> >   This review focused on quality assessment, not line-by-line code audit.
> >   Key code paths and quality issues were examined with surrounding
> context
> >   when concerns arose. Documentation files were reviewed with AI-assisted
> >   grammar and typo checking. Code coverage was measured using gcov and
> >   custom analysis tools.
> >
> > Limitations:
> >   - Not a security audit or formal verification
> >   - Parser and executor internals reviewed at module level, not
> exhaustively
> >   - Performance characteristics not benchmarked
> >
> >
> > TABLE OF CONTENTS
> > -----------------
> >
> > 1. Executive Summary
> > 2. Issues Found
> >    2.1 Critical / Major
> >    2.2 Minor Issues (Review Needed)
> >    2.3 Minor Issues (Style)
> >    2.4 Suggestions for Discussion
> > 3. Test Coverage
> >    3.1 Covered Areas
> >    3.2 Untested Items
> >    3.3 Unimplemented Features (No Test Needed)
> > 4. Code Coverage Analysis
> >    4.1 Overall Coverage
> >    4.2 Coverage by File
> >    4.3 Uncovered Code Risk Assessment
> > 5. Commit Analysis
> > 6. Recommendations
> > 7. Conclusion
> >
> >
> > 1. EXECUTIVE SUMMARY
> > --------------------
> >
> > Overall assessment: GOOD
> >
> > The SQL/RPR patch demonstrates solid implementation quality within its
> > defined scope. Code follows PostgreSQL coding standards (with minor style
> > issues), test coverage is comprehensive at 96.4%, and documentation is
> > thorough with only minor typos.
> >
> > Rating by Area:
> > - Code Quality:     Good (PostgreSQL style compliant, 26 static style
> fixes
> > recommended)
> > - Test Coverage:    Good (96.4% line coverage, 28 test cases)
> > - Documentation:    Good (Complete, 1 minor typo)
> > - Build/Regress:    Pass (make check-world, 753 tests passed)
> >
> >
> > 2. ISSUES FOUND
> > ---------------
> >
> > 2.1 Critical / Major
> >
> > #1 [Code] Greedy pattern combinatorial explosion
> >    Pattern: PATTERN (A+ B+ C+) with DEFINE A AS TRUE, B AS TRUE, C AS
> TRUE
> >    Impact: Memory exhaustion or infinite wait
> >    Recommendation: Add work_mem-based memory limit (error on exceed)
> >
> >    Evidence - No memory limit in current code:
> >    - nodeWindowAgg.c:5718-5722 string_set_init(): palloc() unconditional
> >    - nodeWindowAgg.c:5741-5750 string_set_add(): set_size *= 2;
> repalloc()
> > unlimited
> >    - nodeWindowAgg.c:5095-5174 generate_patterns(): Triple loop, no limit
> >
> >    Only work_mem usage in RPR (nodeWindowAgg.c:1297):
> >      winstate->buffer = tuplestore_begin_heap(false, false, work_mem);
> >    -> For tuple buffer, unrelated to pattern combination memory
> (StringSet)
> >
> > 2.2 Minor Issues (Review Needed)
> >
> > #1 [Code] nodeWindowAgg.c:5849,5909,5912
> >    pos > NUM_ALPHABETS check - intent unclear, causes error at 28 PATTERN
> > elements
> >
> >    Reproduction:
> >    - PATTERN (A B C ... Z A) (27 elements) -> OK
> >    - PATTERN (A B C ... Z A B) (28 elements) -> ERROR "initial is not
> valid
> > char: b"
> >
> > 2.3 Minor Issues (Style)
> >
> > #1 [Code] nodeWindowAgg.c (25 blocks)
> >    #ifdef RPR_DEBUG -> recommend elog(DEBUG2, ...) or remove
> >
> > #2 [Code] src/backend/executor/nodeWindowAgg.c
> >    static keyword on separate line (26 functions)
> >
> > #3 [Code] src/backend/utils/adt/ruleutils.c
> >    Whitespace typo: "regexp !=NULL" -> "regexp != NULL"
> >
> > #4 [Code] nodeWindowAgg.c:4619
> >    Error message case: "Unrecognized" -> "unrecognized" (PostgreSQL
> style)
> >
> > #5 [Doc] doc/src/sgml/ref/select.sgml:1128
> >    Typo: "maximu" -> "maximum"
> >
> > 2.4 Suggestions for Discussion
> >
> > #1 Incremental matching with streaming NFA redesign
> >    Benefits:
> >    - O(n) time complexity per row (current: exponential in worst case)
> >    - Bounded memory via state merging and context absorption
> >    - Natural extension for OR patterns, {n,m} quantifiers, nested groups
> >    - Enables MEASURES clause with incremental aggregates
> >    - Proven approach in CEP engines (Flink, Esper)
> >
> >
> > 3. TEST COVERAGE
> > ----------------
> >
> > 3.1 Covered Areas
> >
> > - PATTERN clause: +, *, ? quantifiers (line 41, 71, 86)
> > - DEFINE clause: Variable conditions, auto-TRUE for missing (line
> 120-133)
> > - PREV/NEXT functions: Single argument (line 44, 173)
> > - AFTER MATCH SKIP: TO NEXT ROW (line 182), PAST LAST ROW (line 198)
> > - Aggregate integration: sum, avg, count, max, min (line 258-277)
> > - Window function integration: first_value, last_value, nth_value (line
> > 34-35)
> > - JOIN/CTE: JOIN (line 313), WITH (line 324)
> > - View: VIEW creation, pg_get_viewdef (line 390-406)
> > - Error cases: 7 error scenarios (line 409-532)
> > - Large partition: 5000 row smoke test (line 360-387)
> > - ROWS BETWEEN offset: CURRENT ROW AND 2 FOLLOWING (line 244)
> >
> > 3.2 Untested Items
> >
> > Feature                  Priority   Recommendation
> >
> -------------------------------------------------------------------------------
> > 26 variable limit        Medium     Test 26 variables success, 27th
> > variable error
> > NULL value handling      Low        Test PREV(col) where col or previous
> > row is NULL
> >
> > Sample test for 26 variable limit:
> >
> >     -- Should fail with 27th variable (parser error, no table needed)
> >     SELECT * FROM (SELECT 1 AS x) t
> >     WINDOW w AS (
> >       ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
> >       PATTERN (a b c d e f g h i j k l m n o p q r s t u v w x y z aa)
> >       DEFINE a AS TRUE, b AS TRUE, c AS TRUE, d AS TRUE, e AS TRUE, f AS
> > TRUE,
> >              g AS TRUE, h AS TRUE, i AS TRUE, j AS TRUE, k AS TRUE, l AS
> > TRUE,
> >              m AS TRUE, n AS TRUE, o AS TRUE, p AS TRUE, q AS TRUE, r AS
> > TRUE,
> >              s AS TRUE, t AS TRUE, u AS TRUE, v AS TRUE, w AS TRUE, x AS
> > TRUE,
> >              y AS TRUE, z AS TRUE, aa AS TRUE
> >     );
> >     -- ERROR: number of row pattern definition variable names exceeds 26
> >
> > Sample test for NULL handling:
> >
> >     CREATE TEMP TABLE stock_null (company TEXT, tdate DATE, price
> INTEGER);
> >     INSERT INTO stock_null VALUES ('c1', '2023-07-01', 100);
> >     INSERT INTO stock_null VALUES ('c1', '2023-07-02', NULL);  -- NULL in
> > middle
> >     INSERT INTO stock_null VALUES ('c1', '2023-07-03', 200);
> >     INSERT INTO stock_null VALUES ('c1', '2023-07-04', 150);
> >
> >     SELECT company, tdate, price, count(*) OVER w AS match_count
> >     FROM stock_null
> >     WINDOW w AS (
> >       PARTITION BY company
> >       ORDER BY tdate
> >       ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
> >       PATTERN (START UP DOWN)
> >       DEFINE START AS TRUE, UP AS price > PREV(price), DOWN AS price <
> > PREV(price)
> >     );
> >     -- Result: all rows show match_count = 0 (NULL breaks pattern
> matching)
> >
> > 3.3 Unimplemented Features (No Test Needed)
> >
> > Per documentation (select.sgml:1133-1136):
> > - MEASURES clause: Not implemented
> > - SUBSET clause: Not implemented
> > - AFTER MATCH variants: Only SKIP TO NEXT ROW, PAST LAST ROW supported
> >
> > Not in documentation, verified by testing:
> > - {n,m} quantifier: Parser error ("syntax error at or near {")
> > - (A|B) OR pattern: Not supported (parsed as single variable name "a|b")
> > - (A B)+ compound repetition: Parser accepts, but execution returns 0
> > matches
> >
> >
> > 4. CODE COVERAGE ANALYSIS
> > -------------------------
> >
> > 4.1 Overall Coverage
> >
> > 96.4% (732 / 759 lines)
> >
> > 4.2 Coverage by File (major RPR-modified files)
> >
> > nodeWindowAgg.c:    96.6% (560/580) - Pattern matching execution engine
> > parse_clause.c:     96.7% (88/91) - PATTERN/DEFINE analysis
> > ruleutils.c:        93.1% (54/58) - pg_get_viewdef output
> >
> > 4.3 Uncovered Code Risk Assessment
> >
> > Total: 27 lines uncovered
> >
> > Medium Risk (2 items) - Test addition recommended (see section 3.2):
> > - parse_clause.c:4093: transformDefineClause - 27th variable error
> > - nodeWindowAgg.c:5623: get_slots - null_slot for PREV at partition first
> > row
> >
> > Low Risk (25 items) - Defensive code / Unreachable:
> > - nodeWindowAgg.c:3074: attno_map_walker - Parser validates arg count
> > - nodeWindowAgg.c:3137-3138: rpr_navigation_walker - switch default
> > - nodeWindowAgg.c:3566: window_gettupleslot - Before mark position
> > - nodeWindowAgg.c:4289: WinGetFuncArgInFrame - isout flag
> > - nodeWindowAgg.c:4393: WinGetSlotInFrame - Boundary check
> > - nodeWindowAgg.c:4618-4619: row_is_in_reduced_frame - switch default
> > - nodeWindowAgg.c:4697: register_reduced_frame_map - pos < 0 check
> > - nodeWindowAgg.c:5007: search_str_set - NULL return continue
> > - nodeWindowAgg.c:5405: do_pattern_match - Regex compile error
> > - nodeWindowAgg.c:5435,5437-5438: do_pattern_match - Regex exec error
> > - nodeWindowAgg.c:5700: pattern_initial - Variable not found
> > - nodeWindowAgg.c:5776: string_set_get - Index range check
> > - nodeWindowAgg.c:5850: variable_pos_register - a-z range check
> > - nodeWindowAgg.c:5910: variable_pos_fetch - a-z range check
> > - nodeWindowAgg.c:5913: variable_pos_fetch - Index range check
> > - parse_clause.c:3989: transformDefineClause - A_Expr type check
> > - parse_clause.c:4145: transformPatternClause - A_Expr type check
> > - ruleutils.c:6904-6908: get_rule_windowspec - SKIP TO NEXT ROW output
> >
> > Conclusion: Most uncovered code consists of defensive error handling or
> > code unreachable due to parser pre-validation. No security or functional
> > risk.
> >
> >
> > 5. COMMIT ANALYSIS
> > ------------------
> >
> > 8 sequential commits:
> >
> > Commit  Area            Files   +/-         Key Content
> >
> -------------------------------------------------------------------------------
> > 1       Raw Parser      4       +174/-16    gram.y grammar
> (PATTERN/DEFINE)
> > 2       Parse/Analysis  4       +277/-1     parse_clause.c analysis logic
> > 3       Rewriter        1       +109/-0     pg_get_viewdef extension
> > 4       Planner         5       +73/-3      WindowAgg plan node extension
> > 5       Executor        4       +1,942/-11  CORE: Pattern matching engine
> > (+1,850)
> > 6       Docs            3       +192/-7     advanced.sgml,
> func-window.sgml
> > 7       Tests           3       +1,585/-1   rpr.sql (532), rpr.out
> (1,052)
> > 8       typedefs        1       +6/-0       pgindent support
> >
> > Code Change Statistics:
> > - Total files: 25
> > - Lines added: 4,358
> > - Lines deleted: 39
> > - Net increase: +4,319 lines
> >
> >
> > 6. RECOMMENDATIONS
> > ------------------
> >
> > 6.1 Combinatorial Explosion Prevention (Major, Required)
> >
> > Add work_mem-based memory limit for StringSet allocation.
> > Location: string_set_add() in nodeWindowAgg.c:5741-5750
> > Consistent with existing PostgreSQL approach (Hash Join, Sort, etc.)
> >
> > 6.2 Code Review Required (Minor, Decision Needed)
> >
> > Location: nodeWindowAgg.c:5849,5909,5912
> > Issue: pos > NUM_ALPHABETS check intent unclear
> > Current: PATTERN with 28 elements causes error
> > Question: Is 27 element limit intentional?
> >
> > 6.3 Code Style Fixes (Minor)
> >
> > - #ifdef RPR_DEBUG: Use elog(DEBUG2, ...) or remove (25 blocks)
> > - static keyword on separate line: 26 functions to fix
> > - Whitespace: "regexp !=NULL" -> "regexp != NULL"
> > - Error message case: "Unrecognized" -> "unrecognized"
> >
> > 6.4 Documentation Fixes (Minor)
> >
> > - select.sgml: "maximu" -> "maximum"
> >
> > 6.5 Test Additions (Optional)
> >
> > Black-box Tests (Functional):
> >
> > Feature              Test Case
> Priority
> >
> -------------------------------------------------------------------------------
> > Variable limit       26 variables success, 27 error               Medium
> > NULL boundary        PREV at partition first row                  Medium
> >
> > White-box Tests (Coverage) - covered by above black-box tests:
> >
> > File:Line                          Target Branch
> >
> -------------------------------------------------------------------------------
> > parse_clause.c:4093                Limit error branch (Variable limit
> test)
> > nodeWindowAgg.c:5623               null_slot usage (NULL boundary test)
> >
> >
> > 7. CONCLUSION
> > -------------
> >
> > Test Quality: GOOD
> >
> > Core functionality is thoroughly tested with comprehensive error case
> > coverage.
> >
> > The patch is well-implemented within its defined scope. Identified issues
> > include
> > one major concern (combinatorial explosion) and minor style/documentation
> > items.
> >
> > Recommended actions before commit:
> > 1. Add work_mem-based memory limit for pattern combinations (required)
> > 2. Clarify pos > NUM_ALPHABETS check intent (review needed)
> > 3. Fix code style issues (optional but recommended)
> > 4. Fix documentation typo (trivial)
> > 5. Add tests for variable limit and NULL handling (optional)
> >
> > Points for discussion (optional):
> > 6. Incremental matching with streaming NFA redesign
> >
> >
> > Attachment:
> > - coverage.tgz (gcov HTML report, RPR-modified code only)
> >
> >
> > ---
> > End of Report
> >
> > 2025년 9월 24일 (수) PM 7:36, Tatsuo Ishii <ishii@postgresql.org>님이 작성:
> >
> >> Attached are the v33 patches for Row pattern recognition.  The
> >> difference from v32 is that the raw parse tree printing patch is not
> >> included in v33. This is because now that we have
> >> debug_print_raw_parse.
> >>
> >> Best regards,
> >> --
> >> Tatsuo Ishii
> >> SRA OSS K.K.
> >> English: http://www.sraoss.co.jp/index_en/
> >> Japanese:http://www.sraoss.co.jp
> >>
>

Best regards,

Henson

  132. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-02T17:19:40Z

    Hi hackers,

I've been working on an alternative implementation approach for Row
Pattern Recognition (RPR) that addresses some limitations in the
current regex-based patch.

Key differences from the existing approach:

1. Streaming NFA instead of regex engine
   - Process rows incrementally without accumulating encoded strings
   - Avoids O(V^N) combinatorial explosion in multi-variable scenarios

2. No 26-variable limit
   - Variables identified by index, not alphabet encoding

3. Proper Lexical Order support
   - Respects PATTERN alternative order for ONE ROW PER MATCH

4. GREEDY/RELUCTANT quantifier support
   - Longest vs shortest match semantics

5. Incremental MEASURES computation
   - Aggregate values computed during matching, no rescan needed

The attached document describes the design in detail, including:
- Data structures (Pattern, MatchState, MatchContext)
- Execution flow and state transitions
- Context absorption optimization for greedy patterns
- AST optimization passes

This is still a work in progress. I'd appreciate any feedback on
the approach before proceeding with PostgreSQL integration.

Best regards,
Henson

------------------------------------------------------------------------
                         Attachment
------------------------------------------------------------------------


========================================================================
      RPR Streaming NFA Pattern Matching System Overview (DRAFT)
========================================================================

0. Motivation and Approach
------------------------------------------------------------------------

0.1 Background: Limitations of PostgreSQL RPR Patch

    The existing PostgreSQL RPR implementation (RPR-base branch) uses
    a regex-based approach:

    ┌─────────────────────────────────────────────────────────────────┐
    │  Existing Implementation (Regex-based)                          │
    ├─────────────────────────────────────────────────────────────────┤
    │                                                                 │
    │  1. Evaluate all DEFINE conditions for each row                 │
    │     → Encode matching variables as alphabets (a, b, c, ...)     │
    │                                                                 │
    │  2. Accumulate encoded string                                   │
    │     "aabbc" (A match, A match, B match, B match, C match)       │
    │                                                                 │
    │  3. Convert PATTERN to regex                                    │
    │     PATTERN (A+ B+ C) → "^a+b+c"                                │
    │                                                                 │
    │  4. Match using PostgreSQL regex engine                         │
    │     pg_regexec(&preg, encoded_str, ...)                         │
    │                                                                 │
    └─────────────────────────────────────────────────────────────────┘

    Limitations (based on nodeWindowAgg.c analysis):

    ┌──────────────────┬──────────────────────────────────────────────┐
    │ Limitation       │ Cause                                        │
    ├──────────────────┼──────────────────────────────────────────────┤
    │ Combinatorial    │ Cartesian product per row in                 │
    │ Explosion O(V^N) │ generate_patterns(). 3 vars, 20 rows → 3.4B  │
    ├──────────────────┼──────────────────────────────────────────────┤
    │ 26 Variable      │ a-z alphabet encoding (NUM_ALPHABETS=26)     │
    │ Limit            │                                              │
    ├──────────────────┼──────────────────────────────────────────────┤
    │ Lexical Order    │ Encoded by DEFINE declaration order,         │
    │ Not Guaranteed   │ PATTERN alternative order ignored.           │
    │                  │ In (B|A) pattern, A encoded first            │
    ├──────────────────┼──────────────────────────────────────────────┤
    │ No RELUCTANT     │ Only greedy flag exists, no shortest match   │
    │ Support          │ logic                                        │
    ├──────────────────┼──────────────────────────────────────────────┤
    │ MEASURES Not     │ Rescan-based workaround, no incremental      │
    │ Implemented      │ aggregation                                  │
    ├──────────────────┼──────────────────────────────────────────────┤
    │ O(N^2) Retry on  │ No intermediate state maintained. On failure │
    │ Match Failure    │ at row N, retry from row K+1 from scratch    │
    └──────────────────┴──────────────────────────────────────────────┘


0.2 Approach: Streaming NFA-based Pattern Matching

    We adopt an NFA-based approach similar to Apache Flink CEP:

    ┌─────────────────────────────────────────────────────────────────┐
    │  NFA-based Approach                                             │
    ├─────────────────────────────────────────────────────────────────┤
    │                                                                 │
    │  Core Ideas:                                                    │
    │    - Compile pattern to flat array (PatternElement[])           │
    │    - Perform state transitions per row (no regex matching)      │
    │    - Merge identical states to prevent redundant computation    │
    │                                                                 │
    │  Time Complexity: O(N x S x E)                                  │
    │    N: Number of input rows                                      │
    │    S: Number of concurrent active states (bounded by merge)     │
    │    E: Number of pattern elements                                │
    │                                                                 │
    └─────────────────────────────────────────────────────────────────┘

    Resolving Limitations:

    ┌──────────────────┬──────────────────────────────────────────────┐
    │ Original Limit   │ Resolution with Streaming NFA                │
    ├──────────────────┼──────────────────────────────────────────────┤
    │ Combinatorial    │ State transition based, identical state merge│
    │ Explosion O(V^N) │ → O(N×S×E), S proportional to pattern size   │
    ├──────────────────┼──────────────────────────────────────────────┤
    │ 26 Variable      │ Integer varId, size adjustable as needed     │
    │ Limit            │                                              │
    ├──────────────────┼──────────────────────────────────────────────┤
    │ Lexical Order    │ #ALT branches generated in PATTERN order     │
    │ Not Guaranteed   │ → Alternative order preserved in paths[][]   │
    ├──────────────────┼──────────────────────────────────────────────┤
    │ No RELUCTANT     │ Custom NFA enables shortest/longest match    │
    │ Support          │ control                                      │
    ├──────────────────┼──────────────────────────────────────────────┤
    │ MEASURES Not     │ Incremental aggregation computes SUM, COUNT  │
    │ Implemented      │ etc. in real-time                            │
    ├──────────────────┼──────────────────────────────────────────────┤
    │ O(N^2) Retry on  │ Multiple Contexts maintained concurrently    │
    │ Match Failure    │ → Parallel tracking per start point, O(N)    │
    └──────────────────┴──────────────────────────────────────────────┘

    New Advantages:

    ┌──────────────────┬──────────────────────────────────────────────┐
    │ Advantage        │ Description                                  │
    ├──────────────────┼──────────────────────────────────────────────┤
    │ ALL ROWS Mode    │ Track all possible match paths concurrently  │
    │                  │ → Each Context branches/completes separately │
    ├──────────────────┼──────────────────────────────────────────────┤
    │ Context          │ Merge Contexts upon reaching identical state │
    │ Absorption       │ → Eliminate redundant computation, memory    │
    ├──────────────────┼──────────────────────────────────────────────┤
    │ Incremental      │ Compute SUM, COUNT etc. during matching      │
    │ Aggregation      │ → No rescan needed after completion          │
    ├──────────────────┼──────────────────────────────────────────────┤
    │ Flat Array       │ Contiguous memory layout for states[],       │
    │ Structure        │ contexts[] → Good cache locality, no pointer │
    ├──────────────────┼──────────────────────────────────────────────┤
    │ Path Sharing     │ Chunk tree + hash table for path sharing     │
    │                  │ → O(1) reference on fork, memory dedup       │
    └──────────────────┴──────────────────────────────────────────────┘


0.3 Difference from Flink: History Elimination

    ┌─────────────────────────────────────────────────────────────────┐
    │  Flink CEP (Streaming)              PostgreSQL (Batch)          │
    ├─────────────────────────────────────────────────────────────────┤
    │                                                                 │
    │  Original data flows away           Materialized data           │
    │  → Cannot re-access                 → Rescan anytime            │
    │  → Store row data in History        → No History needed         │
    │  → Pointer tree + ref counting      → Store match_start/end only│
    │                                                                 │
    │  Fork: pointer copy O(1)            Fork: copy counts[]         │
    │  MEASURES: backtrack History        MEASURES: rescan range      │
    │                                                                 │
    └─────────────────────────────────────────────────────────────────┘

    Our Implementation Choices:
      - Adopt Incremental Aggregation
      - Compute SUM, COUNT etc. in real-time during matching
      - Maintain only aggregate values without History pointers
      - Store path info as varId arrays in paths[][]


0.4 Design Goals

    ┌──────────────┬──────────────────────────────────────────────────┐
    │ Goal         │ Description                                      │
    ├──────────────┼──────────────────────────────────────────────────┤
    │ Linear       │ O(N×S×E), no combinatorial explosion             │
    │ Complexity   │                                                  │
    ├──────────────┼──────────────────────────────────────────────────┤
    │ Standard     │ SQL:2016 RPR semantics (GREEDY, RELUCTANT, SKIP) │
    │ Compliance   │                                                  │
    ├──────────────┼──────────────────────────────────────────────────┤
    │ Extensibility│ Future extensions: MEASURES, SUBSET, etc.        │
    ├──────────────┼──────────────────────────────────────────────────┤
    │ Simplicity   │ Flat array pattern, clear state transition rules │
    └──────────────┴──────────────────────────────────────────────────┘


0.5 Path Storage Optimization

    Risk of memory explosion when paths[][] are copied on Context fork.
    Share identical paths via chunk tree + hash table:

    ┌─────────────────────────────────────────────────────────────────┐
    │  Chunk Tree Structure                                           │
    ├─────────────────────────────────────────────────────────────────┤
    │                                                                 │
    │  Chunk: fixed-size(2) array + parent pointer + ref count        │
    │                                                                 │
    │  Example: Two paths [A,A,C,D] and [A,A,C,E]                     │
    │                                                                 │
    │      Chunk1[A,A] ← Chunk2[C,D]  (Path1)                         │
    │         RC:2    ↖                                               │
    │                   Chunk3[C,E]  (Path2)                          │
    │                                                                 │
    │      → Share parent chunk (Chunk1), create new chunk from fork  │
    │                                                                 │
    │  Hash table: (parent, values) → reuse identical chunks          │
    │                                                                 │
    └─────────────────────────────────────────────────────────────────┘


1. System Architecture
------------------------------------------------------------------------

┌────────────────────────────────────────────────────────────────────────┐
│                        Processing Pipeline                             │
├────────────────────────────────────────────────────────────────────────┤
│                                                                        │
│  Pattern String   Parser      Pattern          Executor         Result │
│  "A+ B | C" ───▶ (AST) ───▶ (elements) ───▶ (Contexts) ───▶ Match  │
│                                                                        │
│  ┌──────────────────┐  ┌───────────────────┐  ┌───────────────────┐    │
│  │      Parser      │  │      Context      │  │     Executor      │    │
│  │                  │  │                   │  │                   │    │
│  │  Tokenize        │  │  MatchState       │  │  NFAExecutor      │    │
│  │  Build AST       │  │  MatchContext     │  │  Multi-Context    │    │
│  │  Optimize        │  │  State Transition │  │  Absorb/SKIP      │    │
│  │  Compile         │  │  Greedy/Reluctant │  │  Output Mgmt      │    │
│  └──────────────────┘  └───────────────────┘  └───────────────────┘    │
│                                                                        │
└────────────────────────────────────────────────────────────────────────┘


2. Document Structure
------------------------------------------------------------------------

┌──────────────┬────────────────────────────────────────────────────┐
│ Parser       │ Pattern string → PatternElement[] transformation   │
│              │  - Tokenizer: string → tokens                      │
│              │  - Parser: tokens → AST                            │
│              │  - Optimizer: AST simplification                   │
│              │  - Compiler: AST → PatternElement[]                │
├──────────────┼────────────────────────────────────────────────────┤
│ Context      │ Single Context internal behavior                   │
│              │  - MatchState: current position + counters + path  │
│              │  - MatchContext: collection of States              │
│              │  - State transition: variable match, #ALT, #END,   │
│              │    #FIN handling                                   │
│              │  - Greedy/Reluctant quantifier behavior            │
├──────────────┼────────────────────────────────────────────────────┤
│ Executor     │ Multi-Context management                           │
│              │  - NFAExecutor: main execution engine              │
│              │  - Context creation/absorption/removal             │
│              │  - SKIP modes (PAST LAST / TO NEXT)                │
│              │  - Output rows (ONE ROW / ALL ROWS)                │
├──────────────┼────────────────────────────────────────────────────┤
│ Improvements │ Future enhancements                                │
│              │  - Path storage optimization (chunk tree + hash)   │
└──────────────┴────────────────────────────────────────────────────┘


3. Key Data Structures
------------------------------------------------------------------------

3.1 Pattern
    The entire compiled pattern.

    Fields:
      - maxDepth: maximum nesting depth
      - elements[]: PatternElement array
      - variables[]: variable name array (variables[varId] → name)
      - firstMatchIsGreedy: whether first match is Greedy
                            (Context absorption condition)

3.2 PatternElement
    A single element of the compiled pattern.

    Fields:
      - varId: variable identifier or special marker (0+ variable,
               negative special)
      - reluctant: flag (true = shortest, false = longest)
      - min, max: quantifier range
      - depth: nesting depth
      - next: next element index
      - jump: alternative/repeat jump target

3.3 MatchState
    A single state during NFA execution.

    Fields:
      - elementIndex: current PatternElement position
      - counts[]: per-depth repeat counters
      - summaries[]: Summary array (creation order preserved)

    State equivalence: identical if elementIndex + counts[] match
    On merge, summaries are combined

3.4 Summary
    Aggregate values and path information.

    Fields:
      - aggregates{}: incremental aggregates (SUM, COUNT, FIRST,
                      LAST, MIN, MAX)
      - paths[][]: matched variable paths (creation order preserved)

    Merge rules:
      - If aggregate values identical → merge Summaries, combine paths
      - summaries[0].paths[0] = Lexical Order priority path

3.5 MatchContext
    Collection of States with the same start point.

    Fields:
      - matchStart: match start row (unique identifier)
      - matchEnd: match end row (stores currentRow when matchedState
                  updated)
      - states[]: active States (creation order preserved, all branch
                  paths)
      - matchedState: MatchState | null (for Greedy fallback)

3.6 NFAExecutor
    Main execution engine.

    Fields:
      - pattern: compiled pattern
      - contexts[]: active Contexts (creation order preserved)
      - completedContexts[]: completed Contexts (sorted by start,
                             pending emit)
      - skipMode: PAST_LAST / TO_NEXT
      - currentRow: currently processing row number
      - history[]: per-row execution snapshots (for debugging)


4. Execution Flow
------------------------------------------------------------------------

┌─────────────────────────────────────────────────────────────────────┐
│                         Per-Row Processing                          │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  FOR EACH row IN data:                                              │
│      │                                                              │
│      ├─▶ 1. tryStartNewContext(row)                                │
│      │       └─ Create Context if new match can start               │
│      │                                                              │
│      ├─▶ 2. FOR EACH context:                                      │
│      │       └─ processContext(context, row)                        │
│      │           └─ State transitions + incremental aggregation     │
│      │                                                              │
│      ├─▶ 3. mergeStates() + absorbContexts()                       │
│      │       ├─ State Merge (same elementIndex+counts)              │
│      │       ├─ Summary Merge (same aggregates → combine paths)     │
│      │       └─ Absorb later Context into earlier one               │
│      │                                                              │
│      ├─▶ 4. Remove dead/completed Contexts                         │
│      │                                                              │
│      └─▶ 5. emitRows()                                             │
│              ├─ contexts[1+] complete → queue in completedContexts  │
│              └─ contexts[0] complete → emit immediately, process    │
│                  queue                                              │
│                  └─ Iterate items where start < contexts[0].start   │
│                      ├─ PAST LAST: discard if start <= prev end     │
│                      └─ TO NEXT: end >= ctx[0].start → stop (wait)  │
│                                  end < ctx[0].start → emit          │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘


5. Core Concepts
------------------------------------------------------------------------

5.1 State Transitions

    Variable element:  If condition true, move to next, add to paths
    #ALT:              Branch to all alternatives (next, jump chain)
    #END:              Check counter, repeat (jump) or escape (next)
    #FIN:              Match complete

5.2 Matching Modes (3 Axes)

    ┌──────────────────────────────────────────────────────────────────┐
    │  [Axis 1] Output Rows:    ONE ROW / ALL ROWS                     │
    │  [Axis 2] Quantifier:     GREEDY / RELUCTANT                     │
    │  [Axis 3] SKIP Option:    PAST LAST / TO NEXT                    │
    └──────────────────────────────────────────────────────────────────┘

    Output Rows:
      - ONE ROW:   Output paths[0] (Lexical Order priority path)
      - ALL ROWS:  Output all paths

    Quantifier Greediness:
      - GREEDY:    Longest match first, preserve completion and continue
      - RELUCTANT: Shortest match first, return immediately on completion

    SKIP Option:
      - PAST LAST: Multiple Contexts active, discard overlaps on emit
      - TO NEXT:   Start possible every row, allow overlapping matches

5.3 Merge Rules

    State Merge (within Context):
      - Condition: same elementIndex + counts[]
      - Action: combine summaries (preserve creation order)
      - Result: reduce State count at same position, prevent redundant
                computation

    Summary Merge (within State):
      - Condition: same aggregates{} values
      - Action: combine paths (preserve creation order)
      - Result: summaries[0].paths[0] = Lexical Order priority path

5.4 Context Absorption

    Condition: First match in pattern is Greedy (max=Infinity AND
               reluctant=false)
               (same elementIndex AND earlier.counts >= later.counts)
    Action: Remove later Context
    Result: Earlier continues with longer match, prevent redundant
            computation

5.5 Emit Flow

    - contexts[0] complete → emit immediately
    - contexts[1+] complete → queue in completedContexts
    - After contexts[0] emits, process queue (by start order):
        1. start >= contexts[0].start → stop (not yet eligible)
        2. PAST LAST: start <= previous emit's end → discard, continue
        3. TO NEXT: end >= contexts[0].start (overlaps with ctx[0])
                    → stop (this item waits until ctx[0] clears)
        4. TO NEXT: end < contexts[0].start (no overlap)
                    → emit, continue to next item


6. Module Dependencies
------------------------------------------------------------------------

  Parser ────────────────────────────────────────────────────────────
    Output: PatternElement[]
    → Context: State references PatternElement
    → Executor: passed as pattern field

  Context ───────────────────────────────────────────────────────────
    Input: PatternElement[] (read-only)
    Output: MatchContext (completed/in-progress status)
    → Executor: State transitions in processContext()

  Executor ──────────────────────────────────────────────────────────
    Input: Pattern, data rows
    Internal: Context creation/management
    Output: Match results (completedContexts)


7. Context Absorption
------------------------------------------------------------------------

    When the pattern starts with a Greedy quantifier (e.g., A+), later
    Contexts can be absorbed into earlier ones if they reach the same
    state with fewer repetitions. This eliminates redundant computation.

┌─────────────────────────────────────────────────────────────────────┐
│                     Context Absorption                              │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  Pattern: A+ B     (A is Greedy: max=Infinity, reluctant=false)     │
│                                                                     │
│  Row 1: A matched                                                   │
│    contexts[0]: start=1, elementIndex=A, counts=[1]                 │
│                                                                     │
│  Row 2: A matched                                                   │
│    contexts[0]: start=1, elementIndex=A, counts=[2]  ← continue     │
│    contexts[1]: start=2, elementIndex=A, counts=[1]  ← new Context  │
│                                                                     │
│  Absorption check:                                                  │
│    - First element is Greedy? YES (A+)                              │
│    - Same elementIndex? YES (both at A)                             │
│    - earlier.counts[0] >= later.counts[0]? YES (2 >= 1)             │
│    → Absorb: Remove contexts[1]                                     │
│                                                                     │
│  Result:                                                            │
│    contexts[0]: start=1, counts=[2]  (only survivor)                │
│                                                                     │
│  Why: Later Context's path is subset of earlier's longer match      │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

    Condition:
      1. Pattern's first match is Greedy (max=Infinity AND reluctant=false)
         - Simple: A+ B
         - Group:  (A B)+ C  (group repeat is also Greedy)
      2. Both Contexts at same elementIndex
      3. earlier.counts[depth] >= later.counts[depth]

    Action: Remove later Context

    Result: Earlier Context continues with longer match,
            prevents redundant computation


8. State Fork and Merge
------------------------------------------------------------------------

    States fork when multiple paths are possible (e.g., at #ALT or when
    min <= count < max at #END). States merge when they reach identical
    positions, combining their paths while preventing redundant work.

┌─────────────────────────────────────────────────────────────────────┐
│                        State Fork                                   │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  Pattern: A{1,2} B    (A can match 1 or 2 times)                    │
│                                                                     │
│  At #END element (after A matched once, count=1):                   │
│    - count(1) >= min(1)? YES → can escape to B (next)               │
│    - count(1) < max(2)?  YES → can repeat A (jump)                  │
│                                                                     │
│  Fork into two States:                                              │
│    State1: elementIndex=B, counts=[1]     ← escaped (try B)         │
│    State2: elementIndex=A, counts=[1]     ← repeat (try more A)     │
│                                                                     │
│  Both States continue in parallel within same Context               │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

    Fork Trigger:
      - #END element where min <= count < max
      - #ALT element (branch to all alternatives)

    Fork Action: Create new State for each branch path

    Copy-on-Write Optimization:
      - Forked States share summaries[] via refcount
      - Only copied when one branch modifies (adds path/updates aggregate)
      - Reduces memory allocation during fork-heavy patterns


┌─────────────────────────────────────────────────────────────────────┐
│                        State Merge                                  │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  Pattern: (A | B) C                                                 │
│                                                                     │
│  After row matches both A and B conditions:                         │
│    State1: path=[A], elementIndex=C, counts=[1]                     │
│    State2: path=[B], elementIndex=C, counts=[1]                     │
│                                                                     │
│  Merge check:                                                       │
│    - Same elementIndex? YES (both at C)                             │
│    - Same counts[]? YES ([1] == [1])                                │
│    → Merge States                                                   │
│                                                                     │
│  Merged State:                                                      │
│    elementIndex=C, counts=[1]                                       │
│    summaries: [                                                     │
│      { paths: [[A]] },    ← from State1 (creation order)            │
│      { paths: [[B]] }     ← from State2                             │
│    ]                                                                │
│                                                                     │
│  Result: Single State, multiple paths preserved                     │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

    Merge Condition:
      - Same elementIndex
      - Same counts[] array

    Merge Action: Combine summaries (preserve creation order)

    Result: Reduces State count, prevents redundant computation
            summaries[0].paths[0] = Lexical Order priority path


┌─────────────────────────────────────────────────────────────────────┐
│              Summary Array Merge (during State Merge)               │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  Same State = Same Future                                           │
│    States with identical (elementIndex + counts[]) will produce     │
│    identical future paths. Only past paths differ.                  │
│                                                                     │
│  Before State merge:                                                │
│    State1: elementIndex=C, counts=[1]                               │
│            summaries: [{ agg:{sum:10}, paths:[[A,C]] }]             │
│    State2: elementIndex=C, counts=[1]                               │
│            summaries: [{ agg:{sum:20}, paths:[[B,C]] }]             │
│            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~                             │
│            same state = same future                                 │
│                                                                     │
│  After State merge:                                                 │
│    Merged: elementIndex=C, counts=[1]                               │
│            summaries: [                                             │
│              { agg:{sum:10}, paths:[[A,C]] },                       │
│              { agg:{sum:20}, paths:[[B,C]] }                        │
│            ]                                                        │
│                                                                     │
│  Why: Merge States now, they will have identical continuations      │
│       summaries[] combined as array (preserve creation order)       │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘


┌─────────────────────────────────────────────────────────────────────┐
│              Summary Merge (= Path Merge)                           │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  When: Two Summaries have identical aggregates{}                    │
│        → Merge Summaries, which merges their paths                  │
│                                                                     │
│  Before merge:                                                      │
│    Summary1: { aggregates: {sum:10}, paths: [[A,A]] }               │
│    Summary2: { aggregates: {sum:10}, paths: [[A,B]] }               │
│                                                                     │
│  After merge:                                                       │
│    Summary: { aggregates: {sum:10}, paths: [[A,A], [A,B]] }         │
│                                                                     │
│  Why: Same aggregate = same "value", only paths differ              │
│       Merging Summaries automatically combines their paths          │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

    Summary Merge Condition:
      - Same aggregates{} values (SUM, COUNT, etc. all equal)

    Summary Merge Action:
      - Merge two Summaries into one
      - paths[][] arrays combined (preserve creation order)
      - paths[0] = Lexical Order priority path


┌─────────────────────────────────────────────────────────────────────┐
│                     Path Deduplication                              │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  When: Duplicate paths exist after merging                          │
│                                                                     │
│  Before dedup:                                                      │
│    paths: [[A,B,C], [A,B,C], [A,B,D]]                               │
│            ~~~~~~   ~~~~~~                                          │
│            duplicate                                                │
│                                                                     │
│  After dedup:                                                       │
│    paths: [[A,B,C], [A,B,D]]                                        │
│                                                                     │
│  Why: Duplicate paths provide no additional information             │
│       Remove to reduce memory and output redundancy                 │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

    Deduplication Condition:
      - Identical variable sequence

    Deduplication Action:
      - Keep first occurrence (preserve creation order)
      - Discard duplicates


9. Completion Handling Algorithm
------------------------------------------------------------------------

    When a State reaches #FIN (elementIndex = -1), the pattern has matched.
    Whether to finalize or continue depends on quantifier mode. GREEDY
    preserves completion and continues seeking longer matches, while
    RELUCTANT completes immediately with the shortest match.

┌──────────────────────────────────────────────────────────────────────┐
│                    Completion Handling Algorithm                     │
├──────────────────────────────────────────────────────────────────────┤
│                                                                      │
│              Completion State Occurs (elementIndex = -1)             │
│                               │                                      │
│                               ▼                                      │
│                ┌──────────────────────────────┐                      │
│                │   Active states remain AND   │                      │
│                │   can match pattern input?   │                      │
│                └──────────────┬───────────────┘                      │
│                               │                                      │
│            YES ┌──────────────┴─────────────────┐ NO                 │
│                │                                │                    │
│                ▼                                ▼                    │
│   ┌─────────────────────────┐    ┌─────────────────────────────┐     │
│   │    Can continue more    │    │    Cannot continue more     │     │
│   └────────────┬────────────┘    └──────────────┬──────────────┘     │
│                │                                │                    │
│                ▼                                ▼                    │
│   ┌─────────────────────────┐    ┌─────────────────────────────┐     │
│   │  Is GREEDY quantifier?  │    │     Finalize result         │     │
│   └───────────┬─────────────┘    │                             │     │
│               │                  │   If preserved → fallback   │     │
│               │             NO   │   If none → match failed    │     │
│         YES ┌─┴──────────────┐   │   → Output Lexical Order 1  │     │
│             │                │   └─────────────────────────────┘     │
│             ▼                ▼                                       │
│   ┌────────────────────┐  ┌───────────────────────┐                  │
│   │ Preserve, continue │  │  Complete immediately │                  │
│   │   (for fallback)   │  │    (RELUCTANT)        │                  │
│   └────────────────────┘  └───────────────────────┘                  │
│                                                                      │
├──────────────────────────────────────────────────────────────────────┤
│  Preservation Rule (GREEDY):                                         │
│    1. Preserve 1 State with longest completion path                  │
│    2. If same length, choose Lexical Order priority State            │
│    3. Replace when new longer completion occurs                      │
│                                                                      │
│  Preserved Content:                                                  │
│    - state: completed MatchState (elementIndex=-1, ...)              │
│    - matchEnd: currentRow at matchedState update time                │
│                                                                      │
│  Fallback Condition (GREEDY):                                        │
│    - All active states dead (input unmatched)                        │
│    - matchedState exists → finalize with preserved State             │
│    - Use preserved matchEnd (saved at update time)                   │
└──────────────────────────────────────────────────────────────────────┘


10. AST Optimization
------------------------------------------------------------------------

    After parsing, the AST undergoes three optimization passes to simplify
    structure before compilation. These reduce PatternElement count,
    improving execution performance.

┌─────────────────────────────────────────────────────────────────────┐
│              1. unwrapGroups (Unnecessary Group Removal)            │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  Case 1: Remove {1,1} Group wrapper                                 │
│                                                                     │
│    Before:  ((A))                   After:  A                       │
│                                                                     │
│      GROUP {1,1}                      VAR:A {1,1}                   │
│       └── GROUP {1,1}                                               │
│            └── VAR:A {1,1}                                          │
│                                                                     │
│  Case 2: Flatten nested SEQ                                         │
│                                                                     │
│    Before:  (A B) C                 After:  A B C                   │
│                                                                     │
│      SEQ                              SEQ                           │
│       ├── GROUP {1,1}                  ├── VAR:A {1,1}              │
│       │    └── SEQ                     ├── VAR:B {1,1}              │
│       │         ├── VAR:A              └── VAR:C {1,1}              │
│       │         └── VAR:B                                           │
│       └── VAR:C                                                     │
│                                                                     │
│  Case 3: Unwrap single-item SEQ/ALT                                 │
│                                                                     │
│    Before:  SEQ[A]                  After:  A                       │
│    Before:  ALT[A]                  After:  A                       │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘


┌─────────────────────────────────────────────────────────────────────┐
│              2. removeDuplicates (Duplicate Alternative Removal)    │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  Before:  A | B | A                 After:  A | B                   │
│                                                                     │
│    ALT                                ALT                           │
│     ├── VAR:A {1,1}                    ├── VAR:A {1,1}              │
│     ├── VAR:B {1,1}                    └── VAR:B {1,1}              │
│     └── VAR:A {1,1}  ← duplicate                                    │
│                                                                     │
│  Comparison: astEqual() for structural equality                     │
│                                                                     │
│  Before:  (A B) | (A B) | C         After:  (A B) | C               │
│                                                                     │
│    ALT                                ALT                           │
│     ├── SEQ[A,B]                       ├── SEQ[A,B]                 │
│     ├── SEQ[A,B]  ← duplicate          └── VAR:C                    │
│     └── VAR:C                                                       │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘


┌─────────────────────────────────────────────────────────────────────┐
│              3. optimizeQuantifiers (Quantifier Optimization)       │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  Case 1: Merge consecutive identical variables                      │
│                                                                     │
│    Before:  A A A B                 After:  A{3} B                  │
│                                                                     │
│      SEQ                              SEQ                           │
│       ├── VAR:A {1,1}                  ├── VAR:A {3,3}              │
│       ├── VAR:A {1,1}                  └── VAR:B {1,1}              │
│       ├── VAR:A {1,1}                                               │
│       └── VAR:B {1,1}                                               │
│                                                                     │
│  Case 2: Merge nested quantifiers (when one is fixed)               │
│                                                                     │
│    Before:  (A{2}){3}               After:  A{6}                    │
│                                                                     │
│      GROUP {3,3}                      VAR:A {6,6}                   │
│       └── VAR:A {2,2}                                               │
│                                                                     │
│    Calculation: min = 2*3 = 6, max = 2*3 = 6                        │
│                                                                     │
│  Note: Only safe when inner or outer quantifier has min == max      │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

    Optimization Order:
      1. unwrapGroups    - Remove unnecessary structure
      2. removeDuplicates - Eliminate redundant alternatives
      3. optimizeQuantifiers - Merge quantifiers

    Combined Example:

      Input: "((A | A) B B)"

      Step 1 (unwrapGroups):
        GROUP{1,1} -> SEQ, inner GROUP{1,1} -> ALT
        Result: SEQ[ALT[A,A], B, B]

      Step 2 (removeDuplicates):
        ALT[A,A] -> A
        Result: SEQ[A, B, B]

      Step 3 (optimizeQuantifiers):
        B B -> B{2}
        Result: SEQ[A, B{2}]


11. AST to PatternElement[] Compilation
------------------------------------------------------------------------

    The optimized AST is flattened into a linear PatternElement array.
    Each element contains pointers (next, jump) for navigation and depth
    for counter indexing. This flat structure enables efficient state
    transitions during NFA execution.

┌─────────────────────────────────────────────────────────────────────┐
│                    AST Flattening Example                           │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  Pattern: "(A | B)+ C"                                              │
│                                                                     │
│  AST:                          PatternElement[]:                    │
│                                                                     │
│    SEQ                         [0] #ALT  next:1              d:1    │
│     ├── GROUP {1,∞}            [1] A     next:3, jump:2      d:2    │
│     │    └── ALT               [2] B     next:3, jump:-1     d:2    │
│     │         ├── VAR:A        [3] #END  next:4, jump:0      d:0    │
│     │         └── VAR:B                  min:1, max:∞               │
│     └── VAR:C {0,1}            [4] C     next:5, min:0, max:1 d:0   │
│                                [5] #FIN  next:-1                    │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

    Pointer Rules:

      next pointer:
        - Sequential elements: index of next element
        - Last element before #FIN: points to #FIN
        - #FIN: -1 (terminal)

      jump pointer:
        - #END: group start index (loop back)
        - ALT branch first element: next alternative start
        - Last alternative: -1

      depth:
        - Top level: 0
        - Inside GROUP/ALT: parent depth + 1
        - #END: parent depth (for repeat counter indexing)

    Flattening Process:

      1. Traverse AST recursively
      2. For each node type:
         - VAR: emit PatternElement with varId, min, max, depth
         - GROUP: flatten content, emit #END with min/max/jump
         - ALT: emit #ALT, flatten each alternative with jump chain
         - SEQ: flatten each item sequentially
      3. Emit #FIN at end
      4. Resolve next pointers (forward references)


┌─────────────────────────────────────────────────────────────────────┐
│                    Pointer Resolution Example                       │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  Pattern: "A | B | C"                                               │
│                                                                     │
│  [0] #ALT  next:1, jump:-1                                          │
│       │                                                             │
│       ├──▶ [1] A  next:4, jump:2  ──┐                              │
│       │                              │                              │
│       ├──▶ [2] B  next:4, jump:3  ──┤ (all point to #FIN)          │
│       │                              │                              │
│       └──▶ [3] C  next:4, jump:-1 ──┘                              │
│                                                                     │
│  [4] #FIN  next:-1                                                  │
│                                                                     │
│  #ALT.next = first branch (A)                                       │
│  Each branch.jump = next branch (or -1 for last)                    │
│  Each branch.next = element after ALT (#FIN)                        │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

  133. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-03T01:04:54Z

    Hi Henson,

Thank you for the detailed design documentation. While learning the
document, just a quick comment and question.

> Hi hackers,
> 
> I've been working on an alternative implementation approach for Row
> Pattern Recognition (RPR) that addresses some limitations in the
> current regex-based patch.
> 
> Key differences from the existing approach:
> 
> 1. Streaming NFA instead of regex engine
>    - Process rows incrementally without accumulating encoded strings
>    - Avoids O(V^N) combinatorial explosion in multi-variable scenarios

Sounds good. As you already pointed out, current approach has a memory
space problem. If Streaming NFA could solve the issue, it would be
really good.

> 2. No 26-variable limit
>    - Variables identified by index, not alphabet encoding

Sounds good.

> 3. Proper Lexical Order support
>    - Respects PATTERN alternative order for ONE ROW PER MATCH

Here do you have "ONE ROW PER MATCH" option of RPR in your mind? In my
understanding it's in MATCH_RECOGNIZE clause, but not in RPR in Window
clause. (ALL ROWS PER MATCH is not applicable either in RPR in
Window clause).

> 4. GREEDY/RELUCTANT quantifier support
>    - Longest vs shortest match semantics

Sounds good.

> 5. Incremental MEASURES computation
>    - Aggregate values computed during matching, no rescan needed
> 
> The attached document describes the design in detail, including:
> - Data structures (Pattern, MatchState, MatchContext)
> - Execution flow and state transitions
> - Context absorption optimization for greedy patterns
> - AST optimization passes
> 
> This is still a work in progress. I'd appreciate any feedback on
> the approach before proceeding with PostgreSQL integration.

I understand you are still in the design phase and sorry for jumping
into an implementation question. but If you have an idea, please
advice:

How do you handle '{' and '}' in the PATTERN clause in the raw parser?
I am not a parser expert but it seems it's not easy to handle '{' and
'}' in gram.y.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  134. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-03T11:11:06Z

    Hi Ishii-san,

Thank you for the quick feedback and helpful spec clarifications!

2026년 1월 3일 (토) AM 10:05, Tatsuo Ishii <ishii@postgresql.org>님이 작성:

> Hi Henson,
>
> Thank you for the detailed design documentation. While learning the
> document, just a quick comment and question.
>
> > Hi hackers,
> >
> > I've been working on an alternative implementation approach for Row
> > Pattern Recognition (RPR) that addresses some limitations in the
> > current regex-based patch.
> >
> > Key differences from the existing approach:
> >
> > 1. Streaming NFA instead of regex engine
> >    - Process rows incrementally without accumulating encoded strings
> >    - Avoids O(V^N) combinatorial explosion in multi-variable scenarios
>
> Sounds good. As you already pointed out, current approach has a memory
> space problem. If Streaming NFA could solve the issue, it would be
> really good.
>
> > 2. No 26-variable limit
> >    - Variables identified by index, not alphabet encoding
>
> Sounds good.
>
> > 3. Proper Lexical Order support
> >    - Respects PATTERN alternative order for ONE ROW PER MATCH
>
> Here do you have "ONE ROW PER MATCH" option of RPR in your mind? In my
> understanding it's in MATCH_RECOGNIZE clause, but not in RPR in Window
> clause. (ALL ROWS PER MATCH is not applicable either in RPR in
> Window clause).
>
>
You're absolutely right - thank you for the correction!

I'm currently exploring what this NFA structure could theoretically
support, rather than strictly binding to the spec yet. Since I'm
still learning from Oracle manuals and your code without access to
the full SQL:2016 standard, I mixed up MATCH_RECOGNIZE syntax with
Window clause RPR - sorry for the confusion!

Once I have a clearer understanding of the spec, I'll properly
define how it integrates with PostgreSQL's grammar.

Your spec knowledge is invaluable as I continue learning. Thank you!


> > 4. GREEDY/RELUCTANT quantifier support
> >    - Longest vs shortest match semantics
>
> Sounds good.
>
> > 5. Incremental MEASURES computation
> >    - Aggregate values computed during matching, no rescan needed
> >
> > The attached document describes the design in detail, including:
> > - Data structures (Pattern, MatchState, MatchContext)
> > - Execution flow and state transitions
> > - Context absorption optimization for greedy patterns
> > - AST optimization passes
> >
> > This is still a work in progress. I'd appreciate any feedback on
> > the approach before proceeding with PostgreSQL integration.
>
> I understand you are still in the design phase and sorry for jumping
> into an implementation question. but If you have an idea, please
> advice:
>
>
Not at all - implementation questions are valuable advice that
help catch what I might miss in the design phase!


> How do you handle '{' and '}' in the PATTERN clause in the raw parser?
> I am not a parser expert but it seems it's not easy to handle '{' and
> '}' in gram.y.
>
>
You're right - it's not straightforward. Both gram.y and scan.l
need to be modified together.

I've attached a rough patch showing one possible approach. However,
since the OR handling has also been modified in SQL/PGQ, there will
likely be conflicts that need to be resolved when integrating.


> Best regards,
> --
> Tatsuo Ishii
> SRA OSS K.K.
> English: http://www.sraoss.co.jp/index_en/
> Japanese:http://www.sraoss.co.jp
>

  135. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-04T12:28:25Z

    Hi Henson,

>> Here do you have "ONE ROW PER MATCH" option of RPR in your mind? In my
>> understanding it's in MATCH_RECOGNIZE clause, but not in RPR in Window
>> clause. (ALL ROWS PER MATCH is not applicable either in RPR in
>> Window clause).
>>
>>
> You're absolutely right - thank you for the correction!
> 
> I'm currently exploring what this NFA structure could theoretically
> support, rather than strictly binding to the spec yet. Since I'm
> still learning from Oracle manuals and your code without access to
> the full SQL:2016 standard, I mixed up MATCH_RECOGNIZE syntax with
> Window clause RPR - sorry for the confusion!

No problem at all. MATCH_RECOGNIZE and Window clause RPR are quite
similar. In the mean time "Trino" manual maybe useful for you if you
don't have access to the SQL standard.
https://trino.io/docs/current/sql/pattern-recognition-in-window.html

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  136. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-04T14:01:44Z

    Hi Henson,

Thank you for the comprehensive review!

> SQL/RPR Patch Review Report
> ============================
> 
> Patch: Row Pattern Recognition (SQL:2016)
> Commitfest: https://commitfest.postgresql.org/patch/4460
> 
> Review Methodology:
>   This review focused on quality assessment, not line-by-line code audit.
>   Key code paths and quality issues were examined with surrounding context
>   when concerns arose. Documentation files were reviewed with AI-assisted
>   grammar and typo checking. Code coverage was measured using gcov and
>   custom analysis tools.
> 
> Limitations:
>   - Not a security audit or formal verification
>   - Parser and executor internals reviewed at module level, not exhaustively
>   - Performance characteristics not benchmarked
> 
> 
> TABLE OF CONTENTS
> -----------------
> 
> 1. Executive Summary
> 2. Issues Found
>    2.1 Critical / Major
>    2.2 Minor Issues (Review Needed)
>    2.3 Minor Issues (Style)
>    2.4 Suggestions for Discussion
> 3. Test Coverage
>    3.1 Covered Areas
>    3.2 Untested Items
>    3.3 Unimplemented Features (No Test Needed)
> 4. Code Coverage Analysis
>    4.1 Overall Coverage
>    4.2 Coverage by File
>    4.3 Uncovered Code Risk Assessment
> 5. Commit Analysis
> 6. Recommendations
> 7. Conclusion
> 
> 
> 1. EXECUTIVE SUMMARY
> --------------------
> 
> Overall assessment: GOOD

Glad to hear that.

> The SQL/RPR patch demonstrates solid implementation quality within its
> defined scope. Code follows PostgreSQL coding standards (with minor style
> issues), test coverage is comprehensive at 96.4%, and documentation is
> thorough with only minor typos.
> 
> Rating by Area:
> - Code Quality:     Good (PostgreSQL style compliant, 26 static style fixes
> recommended)
> - Test Coverage:    Good (96.4% line coverage, 28 test cases)
> - Documentation:    Good (Complete, 1 minor typo)
> - Build/Regress:    Pass (make check-world, 753 tests passed)
> 
> 
> 2. ISSUES FOUND
> ---------------
> 
> 2.1 Critical / Major
> 
> #1 [Code] Greedy pattern combinatorial explosion
>    Pattern: PATTERN (A+ B+ C+) with DEFINE A AS TRUE, B AS TRUE, C AS TRUE
>    Impact: Memory exhaustion or infinite wait
>    Recommendation: Add work_mem-based memory limit (error on exceed)

Not sure it's a good solution. I think it's acceptable for users that
the query execution getting slow down when there's not enough
work_mem, but ending up with ERROR would not be acceptable. In order
to run the query without ERROR (and the query getting slow) when
there's not enough memory, RPR needs to use temporary files. Problem
is, if the access pattern is random, accessing files in random manner
would be extremely slow. Unfortunately RPR's pattern match access
pattern is surely random in the current patch.  Thus I think adding
memory usage check is not worth the trouble. What do you think?

Instead we need to look for alternative algorithm to perform the
search.  I hope streaming NFA is a good candidate.

>    Evidence - No memory limit in current code:
>    - nodeWindowAgg.c:5718-5722 string_set_init(): palloc() unconditional
>    - nodeWindowAgg.c:5741-5750 string_set_add(): set_size *= 2; repalloc()
> unlimited
>    - nodeWindowAgg.c:5095-5174 generate_patterns(): Triple loop, no limit
> 
>    Only work_mem usage in RPR (nodeWindowAgg.c:1297):
>      winstate->buffer = tuplestore_begin_heap(false, false, work_mem);
>    -> For tuple buffer, unrelated to pattern combination memory (StringSet)
>
> 2.2 Minor Issues (Review Needed)
> 
> #1 [Code] nodeWindowAgg.c:5849,5909,5912
>    pos > NUM_ALPHABETS check - intent unclear, causes error at 28 PATTERN
> elements

Yeah, in variable_pos_fetch() and variable_pos_register(), following
lines are wrong.
	if (pos < 0 || pos > NUM_ALPHABETS)
	if (index < 0 || index > NUM_ALPHABETS)

They should have been:
	if (pos < 0 || pos >= NUM_ALPHABETS)
	if (index < 0 || index >= NUM_ALPHABETS)

Will fix.

>    Reproduction:
>    - PATTERN (A B C ... Z A) (27 elements) -> OK
>    - PATTERN (A B C ... Z A B) (28 elements) -> ERROR "initial is not valid
> char: b"
> 
> 2.3 Minor Issues (Style)
> 
> #1 [Code] nodeWindowAgg.c (25 blocks)
>    #ifdef RPR_DEBUG -> recommend elog(DEBUG2, ...) or remove

I replaced each of them to "printf...". This way, each debug line is
not accompanied with a query, which is good for developers.

> #2 [Code] src/backend/executor/nodeWindowAgg.c
>    static keyword on separate line (26 functions)

Fixed.

> #3 [Code] src/backend/utils/adt/ruleutils.c
>    Whitespace typo: "regexp !=NULL" -> "regexp != NULL"

Fixed.

> #4 [Code] nodeWindowAgg.c:4619
>    Error message case: "Unrecognized" -> "unrecognized" (PostgreSQL style)

Fixed.

> #5 [Doc] doc/src/sgml/ref/select.sgml:1128
>    Typo: "maximu" -> "maximum"

Fixed.

> 2.4 Suggestions for Discussion
> 
> #1 Incremental matching with streaming NFA redesign
>    Benefits:
>    - O(n) time complexity per row (current: exponential in worst case)
>    - Bounded memory via state merging and context absorption
>    - Natural extension for OR patterns, {n,m} quantifiers, nested groups
>    - Enables MEASURES clause with incremental aggregates
>    - Proven approach in CEP engines (Flink, Esper)

I looking forward to join the discussion.

> 3. TEST COVERAGE
> ----------------
> 
> 3.1 Covered Areas
> 
> - PATTERN clause: +, *, ? quantifiers (line 41, 71, 86)
> - DEFINE clause: Variable conditions, auto-TRUE for missing (line 120-133)
> - PREV/NEXT functions: Single argument (line 44, 173)
> - AFTER MATCH SKIP: TO NEXT ROW (line 182), PAST LAST ROW (line 198)
> - Aggregate integration: sum, avg, count, max, min (line 258-277)
> - Window function integration: first_value, last_value, nth_value (line
> 34-35)
> - JOIN/CTE: JOIN (line 313), WITH (line 324)
> - View: VIEW creation, pg_get_viewdef (line 390-406)
> - Error cases: 7 error scenarios (line 409-532)
> - Large partition: 5000 row smoke test (line 360-387)
> - ROWS BETWEEN offset: CURRENT ROW AND 2 FOLLOWING (line 244)
> 
> 3.2 Untested Items
> 
> Feature                  Priority   Recommendation
> -------------------------------------------------------------------------------
> 26 variable limit        Medium     Test 26 variables success, 27th
> variable error
> NULL value handling      Low        Test PREV(col) where col or previous
> row is NULL
> 
> Sample test for 26 variable limit:
> 
>     -- Should fail with 27th variable (parser error, no table needed)
>     SELECT * FROM (SELECT 1 AS x) t
>     WINDOW w AS (
>       ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>       PATTERN (a b c d e f g h i j k l m n o p q r s t u v w x y z aa)
>       DEFINE a AS TRUE, b AS TRUE, c AS TRUE, d AS TRUE, e AS TRUE, f AS
> TRUE,
>              g AS TRUE, h AS TRUE, i AS TRUE, j AS TRUE, k AS TRUE, l AS
> TRUE,
>              m AS TRUE, n AS TRUE, o AS TRUE, p AS TRUE, q AS TRUE, r AS
> TRUE,
>              s AS TRUE, t AS TRUE, u AS TRUE, v AS TRUE, w AS TRUE, x AS
> TRUE,
>              y AS TRUE, z AS TRUE, aa AS TRUE
>     );
>     -- ERROR: number of row pattern definition variable names exceeds 26

Good cacth. I will add the test. BTW, the sample test SQL can be
simplied like this:

SELECT * FROM (SELECT 1 AS x) t
 WINDOW w AS (
 ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
 PATTERN (a b c d e f g h i j k l m n o p q r s t u v w x y z aa)
 DEFINE a AS TRUE
);

Because if a pattern varible (for example "b") is not in the DEFINE
clause, it is treated as if being defined "b AS TRUE" per spec. Same
thing can be said to other pattern variables c ... aa).

> Sample test for NULL handling:
> 
>     CREATE TEMP TABLE stock_null (company TEXT, tdate DATE, price INTEGER);
>     INSERT INTO stock_null VALUES ('c1', '2023-07-01', 100);
>     INSERT INTO stock_null VALUES ('c1', '2023-07-02', NULL);  -- NULL in
> middle
>     INSERT INTO stock_null VALUES ('c1', '2023-07-03', 200);
>     INSERT INTO stock_null VALUES ('c1', '2023-07-04', 150);
> 
>     SELECT company, tdate, price, count(*) OVER w AS match_count
>     FROM stock_null
>     WINDOW w AS (
>       PARTITION BY company
>       ORDER BY tdate
>       ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>       PATTERN (START UP DOWN)
>       DEFINE START AS TRUE, UP AS price > PREV(price), DOWN AS price <
> PREV(price)
>     );
>     -- Result: all rows show match_count = 0 (NULL breaks pattern matching)

Thanks. I will add this test.

> 3.3 Unimplemented Features (No Test Needed)
> 
> Per documentation (select.sgml:1133-1136):
> - MEASURES clause: Not implemented
> - SUBSET clause: Not implemented
> - AFTER MATCH variants: Only SKIP TO NEXT ROW, PAST LAST ROW supported
> 
> Not in documentation, verified by testing:
> - {n,m} quantifier: Parser error ("syntax error at or near {")
> - (A|B) OR pattern: Not supported (parsed as single variable name "a|b")
> - (A B)+ compound repetition: Parser accepts, but execution returns 0
> matches
> 
> 
> 4. CODE COVERAGE ANALYSIS
> -------------------------
> 
> 4.1 Overall Coverage
> 
> 96.4% (732 / 759 lines)
> 
> 4.2 Coverage by File (major RPR-modified files)
> 
> nodeWindowAgg.c:    96.6% (560/580) - Pattern matching execution engine
> parse_clause.c:     96.7% (88/91) - PATTERN/DEFINE analysis
> ruleutils.c:        93.1% (54/58) - pg_get_viewdef output
> 
> 4.3 Uncovered Code Risk Assessment
> 
> Total: 27 lines uncovered
> 
> Medium Risk (2 items) - Test addition recommended (see section 3.2):
> - parse_clause.c:4093: transformDefineClause - 27th variable error
> - nodeWindowAgg.c:5623: get_slots - null_slot for PREV at partition first
> row
> 
> Low Risk (25 items) - Defensive code / Unreachable:
> - nodeWindowAgg.c:3074: attno_map_walker - Parser validates arg count
> - nodeWindowAgg.c:3137-3138: rpr_navigation_walker - switch default
> - nodeWindowAgg.c:3566: window_gettupleslot - Before mark position
> - nodeWindowAgg.c:4289: WinGetFuncArgInFrame - isout flag
> - nodeWindowAgg.c:4393: WinGetSlotInFrame - Boundary check
> - nodeWindowAgg.c:4618-4619: row_is_in_reduced_frame - switch default
> - nodeWindowAgg.c:4697: register_reduced_frame_map - pos < 0 check
> - nodeWindowAgg.c:5007: search_str_set - NULL return continue
> - nodeWindowAgg.c:5405: do_pattern_match - Regex compile error
> - nodeWindowAgg.c:5435,5437-5438: do_pattern_match - Regex exec error
> - nodeWindowAgg.c:5700: pattern_initial - Variable not found
> - nodeWindowAgg.c:5776: string_set_get - Index range check
> - nodeWindowAgg.c:5850: variable_pos_register - a-z range check
> - nodeWindowAgg.c:5910: variable_pos_fetch - a-z range check
> - nodeWindowAgg.c:5913: variable_pos_fetch - Index range check
> - parse_clause.c:3989: transformDefineClause - A_Expr type check
> - parse_clause.c:4145: transformPatternClause - A_Expr type check
> - ruleutils.c:6904-6908: get_rule_windowspec - SKIP TO NEXT ROW output
> 
> Conclusion: Most uncovered code consists of defensive error handling or
> code unreachable due to parser pre-validation. No security or functional
> risk.
> 
> 
> 5. COMMIT ANALYSIS
> ------------------
> 
> 8 sequential commits:
> 
> Commit  Area            Files   +/-         Key Content
> -------------------------------------------------------------------------------
> 1       Raw Parser      4       +174/-16    gram.y grammar (PATTERN/DEFINE)
> 2       Parse/Analysis  4       +277/-1     parse_clause.c analysis logic
> 3       Rewriter        1       +109/-0     pg_get_viewdef extension
> 4       Planner         5       +73/-3      WindowAgg plan node extension
> 5       Executor        4       +1,942/-11  CORE: Pattern matching engine
> (+1,850)
> 6       Docs            3       +192/-7     advanced.sgml, func-window.sgml
> 7       Tests           3       +1,585/-1   rpr.sql (532), rpr.out (1,052)
> 8       typedefs        1       +6/-0       pgindent support
> 
> Code Change Statistics:
> - Total files: 25
> - Lines added: 4,358
> - Lines deleted: 39
> - Net increase: +4,319 lines
> 
> 
> 6. RECOMMENDATIONS
> ------------------
> 
> 6.1 Combinatorial Explosion Prevention (Major, Required)
> 
> Add work_mem-based memory limit for StringSet allocation.
> Location: string_set_add() in nodeWindowAgg.c:5741-5750
> Consistent with existing PostgreSQL approach (Hash Join, Sort, etc.)

See the comment above.

> 6.2 Code Review Required (Minor, Decision Needed)
> 
> Location: nodeWindowAgg.c:5849,5909,5912
> Issue: pos > NUM_ALPHABETS check intent unclear
> Current: PATTERN with 28 elements causes error
> Question: Is 27 element limit intentional?

I think these are addressed in the attached v37 patch.

> 6.3 Code Style Fixes (Minor)
> 
> - #ifdef RPR_DEBUG: Use elog(DEBUG2, ...) or remove (25 blocks)
> - static keyword on separate line: 26 functions to fix
> - Whitespace: "regexp !=NULL" -> "regexp != NULL"
> - Error message case: "Unrecognized" -> "unrecognized"

Fixed in v37.

> 6.4 Documentation Fixes (Minor)
> 
> - select.sgml: "maximu" -> "maximum"

Fixed in v37.

> 6.5 Test Additions (Optional)
> 
> Black-box Tests (Functional):
> 
> Feature              Test Case                                    Priority
> -------------------------------------------------------------------------------
> Variable limit       26 variables success, 27 error               Medium
> NULL boundary        PREV at partition first row                  Medium
> 
> White-box Tests (Coverage) - covered by above black-box tests:
> 
> File:Line                          Target Branch
> -------------------------------------------------------------------------------
> parse_clause.c:4093                Limit error branch (Variable limit test)
> nodeWindowAgg.c:5623               null_slot usage (NULL boundary test)

Test added in the v37 patch.

> 7. CONCLUSION
> -------------
> 
> Test Quality: GOOD
> 
> Core functionality is thoroughly tested with comprehensive error case
> coverage.
> 
> The patch is well-implemented within its defined scope. Identified issues
> include
> one major concern (combinatorial explosion) and minor style/documentation
> items.
> 
> Recommended actions before commit:
> 1. Add work_mem-based memory limit for pattern combinations (required)
> 2. Clarify pos > NUM_ALPHABETS check intent (review needed)
> 3. Fix code style issues (optional but recommended)
> 4. Fix documentation typo (trivial)
> 5. Add tests for variable limit and NULL handling (optional)
> 
> Points for discussion (optional):
> 6. Incremental matching with streaming NFA redesign

Definitely we need discussion on this.

> Attachment:
> - coverage.tgz (gcov HTML report, RPR-modified code only)
> 
> 
> ---
> End of Report

Again thank you for the review. Attached are the v37 patches. This
time I added a short description about the patch to the 001 patch. In
the file description about what are implemented and waht not
implemented. Hope this helps to understand the current status of the
patches.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  137. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-04T15:47:58Z

    Hi Ishii-san,

I've been working on an NFA implementation prototype.

This is a very early prototype - still buggy and incomplete,
but shows the basic approach. Attached HTML file - open in
browser (runs locally, no server needed).

I used JavaScript for quick prototyping and visualization.
The actual PostgreSQL implementation would be in C, but the
algorithm logic remains the same.

To test:
1. Use default pattern
2. Enter variables sequentially: A, B, C, D, E, F, G, H, I
3. Try different patterns and options in the UI

Implemented:
- Patterns: *, +, ?, (A|B), (A B)+
- Reluctant quantifiers: A+?, A*? (switches to reluctant mode)
  Note: This part probably has issues
- Execution trace with state visualization
- Context management: multiple matches, absorption
- Options: SKIP modes, output modes

Many rough edges, especially reluctant matching.
Just sharing the direction.

My responses may be slow due to company work during weekdays.

Best regards,
Henson

  138. Re: Row pattern recognition

    Jacob Champion <jacob.champion@enterprisedb.com> — 2026-01-05T22:07:45Z

    On Sun, Jan 4, 2026 at 7:50 AM Henson Choi <assam258@gmail.com> wrote:
> I've been working on an NFA implementation prototype.

Neat! In case it helps, you may also be interested in my (now very out
of date) mails at [1] and [2].

At one point I was also working on an independent implementation of
the preferment rules, to make it easier for developers to intuit how
matches were made, but I can't find that email at the moment. This may
have some overlap with "state visualization" for the engine? I have a
3k-line patch based on top of v22 of the patchset as of September
2024, if there is interest in that, but no promises on quality -- I
would have posted it if I thought it was ready for review :D

Thanks,
--Jacob

[1] https://postgr.es/m/96b4d3c8-1415-04db-5629-d2617ac7a156%40timescale.com
[2] https://postgr.es/m/CAGu%3Du8gcyAQAsDLp-01R21%3DM4bpAiGP9PZz0OyiMo5_bM30PMA%40mail.gmail.com

  139. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-05T23:26:15Z

    > On Sun, Jan 4, 2026 at 7:50 AM Henson Choi <assam258@gmail.com> wrote:
>> I've been working on an NFA implementation prototype.
> 
> Neat! In case it helps, you may also be interested in my (now very out
> of date) mails at [1] and [2].
> 
> At one point I was also working on an independent implementation of
> the preferment rules, to make it easier for developers to intuit how
> matches were made, but I can't find that email at the moment.

Maybe this?

https://www.postgresql.org/message-id/CAOYmi%2Bns3kHjC83ap_BCfJCL0wfO5BJ_sEByOEpgNOrsPhqQTg%40mail.gmail.com

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  140. Re: Row pattern recognition

    Jacob Champion <jacob.champion@enterprisedb.com> — 2026-01-06T00:11:28Z

    On Mon, Jan 5, 2026 at 3:26 PM Tatsuo Ishii <ishii@postgresql.org> wrote:
> Maybe this?
>
> https://www.postgresql.org/message-id/CAOYmi%2Bns3kHjC83ap_BCfJCL0wfO5BJ_sEByOEpgNOrsPhqQTg%40mail.gmail.com

That's the one, thanks!

--Jacob

  141. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-07T03:21:54Z

    Hi Henson,

Thank you for the design proposal. I'm still learning it, and I have a
few questions for now.

> 3. Proper Lexical Order support
>    - Respects PATTERN alternative order for ONE ROW PER MATCH

RPR in WINDOW clause does not allow to specify "ONE ROW PER MATCH".
(nor ALL ROWS PER MATCH). So I am not sure what you mean here.

> 4. GREEDY/RELUCTANT quantifier support
>    - Longest vs shortest match semantics
> 
> 5. Incremental MEASURES computation
>    - Aggregate values computed during matching, no rescan needed

In my understanding MEASURES does not directly connect to Aggregate
computation with rescan. Can you elaborate why implementing MEASURES
allows to avoid recan for aggregate computation?

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  142. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-10T09:20:16Z

    >
> Neat! In case it helps, you may also be interested in my (now very out
> of date) mails at [1] and [2].
>

 Thanks for the pointers! I'll definitely dig into those threads.


> At one point I was also working on an independent implementation of
> the preferment rules, to make it easier for developers to intuit how
> matches were made, but I can't find that email at the moment. This may
> have some overlap with "state visualization" for the engine? I have a
> 3k-line patch based on top of v22 of the patchset as of September
> 2024, if there is interest in that, but no promises on quality -- I
> would have posted it if I thought it was ready for review :D
>

I'd definitely be interested in seeing it, rough edges and all. I haven't
tackled the PostgreSQL integration design yet, so understanding how
preferment rules map to the existing infrastructure would be very helpful.
(Tatsuo found that email, by the way.)


>
> Thanks,
> --Jacob
>
> [1]
> https://postgr.es/m/96b4d3c8-1415-04db-5629-d2617ac7a156%40timescale.com
> [2]
> https://postgr.es/m/CAGu%3Du8gcyAQAsDLp-01R21%3DM4bpAiGP9PZz0OyiMo5_bM30PMA%40mail.gmail.com


Best regards,
Henson

  143. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-10T09:42:40Z

    Hi Ishii-san,
Thank you for the questions and corrections!

> 3. Proper Lexical Order support
> >    - Respects PATTERN alternative order for ONE ROW PER MATCH
>
> RPR in WINDOW clause does not allow to specify "ONE ROW PER MATCH".
> (nor ALL ROWS PER MATCH). So I am not sure what you mean here.


You're absolutely right to point this out. I've been working without access
to the SQL:2016 standard, relying on Oracle manuals and your implementation,
which led me to incorrectly treat Window clause RPR as a variant of
MATCH_RECOGNIZE.

I now realize there are fundamental differences between R010 (RPR in window
functions) and R020 (MATCH_RECOGNIZE), and I was conflating the two. My
company is supportive of this work, and we're planning to purchase the
standard next week so I can properly understand the spec requirements.

Thank you for catching this - it's exactly the kind of spec guidance I need
as I continue learning.

  > 5. Incremental MEASURES computation

> >    - Aggregate values computed during matching, no rescan needed
>
> In my understanding MEASURES does not directly connect to Aggregate
> computation with rescan. Can you elaborate why implementing MEASURES
> allows to avoid recan for aggregate computation?
>

Let me clarify what I meant by "incremental aggregation" and "rescan":
In the NFA design, I'm building infrastructure for incremental aggregate
computation during pattern matching - maintaining SUM, COUNT, etc. as the
match progresses. When a match completes, if only aggregate functions are
needed, the result can be produced without accessing the original rows
again.

I used "rescan" to contrast this with what I assumed was the existing
approach: match first, then aggregate over the matched row range afterward.
However, I haven't studied your implementation carefully enough to know if
this assumption is correct.

Could you clarify how aggregates are currently computed after pattern
matching
in your implementation? This would help me understand whether the
incremental
approach actually provides a benefit, or if I'm solving a problem that
doesn't
exist.

Regarding MEASURES - I incorrectly connected it to this aggregation
discussion.
As you noted, MEASURES is a separate R020 feature, not part of R010. The
incremental aggregation infrastructure would support both cases, but they're
distinct features.

Best regards,
Henson

  144. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-11T00:41:31Z

    >> 3. Proper Lexical Order support
>> >    - Respects PATTERN alternative order for ONE ROW PER MATCH
>>
>> RPR in WINDOW clause does not allow to specify "ONE ROW PER MATCH".
>> (nor ALL ROWS PER MATCH). So I am not sure what you mean here.
> 
> 
> You're absolutely right to point this out. I've been working without access
> to the SQL:2016 standard, relying on Oracle manuals and your implementation,
> which led me to incorrectly treat Window clause RPR as a variant of
> MATCH_RECOGNIZE.
> 
> I now realize there are fundamental differences between R010 (RPR in window
> functions) and R020 (MATCH_RECOGNIZE), and I was conflating the two. My
> company is supportive of this work, and we're planning to purchase the
> standard next week so I can properly understand the spec requirements.
> 
> Thank you for catching this - it's exactly the kind of spec guidance I need
> as I continue learning.

Glad to hear that you are planning to purchase the standard.  I was
advised by Vik and Jacob the same thing and I really appreciate them
for the suggestion. Without access to 19075-5, I think it's not
possible to implement RPR.

>   > 5. Incremental MEASURES computation
> 
>> >    - Aggregate values computed during matching, no rescan needed
>>
>> In my understanding MEASURES does not directly connect to Aggregate
>> computation with rescan. Can you elaborate why implementing MEASURES
>> allows to avoid recan for aggregate computation?
>>
> 
> Let me clarify what I meant by "incremental aggregation" and "rescan":
> In the NFA design, I'm building infrastructure for incremental aggregate
> computation during pattern matching - maintaining SUM, COUNT, etc. as the
> match progresses. When a match completes, if only aggregate functions are
> needed, the result can be produced without accessing the original rows
> again.

Oh, ok.

> I used "rescan" to contrast this with what I assumed was the existing
> approach: match first, then aggregate over the matched row range afterward.
> However, I haven't studied your implementation carefully enough to know if
> this assumption is correct.

As far as RPR concerns, you are correct. Current implementation does
the "rescan". 

> Could you clarify how aggregates are currently computed after pattern
> matching
> in your implementation? This would help me understand whether the
> incremental
> approach actually provides a benefit, or if I'm solving a problem that
> doesn't
> exist.

Yes, if RPR is enabled, we restart to compute aggregation from the
head of the "reduced frame" (which means the starting row of the
pattern matching), to the end of matching rows. Without RPR,
PostgreSQL reuses the previous aggregate result in some conditions to
avoid the restarting (see the long comment in
eval_windowaggregates()). But currently I think it's not possible for
RPR to avoid it. The reason was explained in the v17 patch (posted on
2024/4/28):

  - In 0005 executor patch, aggregation in RPR always restarts for each
  row. This is necessary to run aggregates on no matching (due to
  skipping) or empty matching (due to no pattern variables matches)
  rows to produce NULL (most aggregates) or 0 (count) properly. In v16
  I had a hack using a flag to force the aggregation results to be
  NULL in case of no match or empty match in
  finalize_windowaggregate(). v17 eliminates the dirty hack.

> Regarding MEASURES - I incorrectly connected it to this aggregation
> discussion.
> As you noted, MEASURES is a separate R020 feature, not part of R010. The
> incremental aggregation infrastructure would support both cases, but they're
> distinct features.

I think R010 has MEASURES too.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  145. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-12T06:44:18Z

    Subject: Re: Row pattern recognition

Hi Ishii-san,

I've been working on the pattern grammar and AST for row pattern
recognition.
The attached patch extends the PATTERN clause to support more regex-like
syntax.

The main additions are:

- Alternation: (A | B)
- Grouping with quantifiers: (A B)+, (A | B)*
- Full quantifier support: +, *, ?, {n}, {n,}, {n,m}
- Reluctant quantifiers: +?, *?, ?? (parsed but not executed yet)

I've introduced RPRPatternNode to represent the pattern as a Thompson-style
AST with four node types: VAR, SEQ, ALT, and GROUP. Each node stores min/max
bounds for quantifiers and a reluctant flag. This should make the eventual
NFA implementation more straightforward.

The parser also does some basic pattern optimizations. For example:
  - (A (B C)) gets flattened to (A B C)
  - ((A)) unwraps to just A
  - (A{2}){3} becomes A{6}
  - (A | B | A) simplifies to (A | B)
  - A A A merges into A{3,3}

I also updated ruleutils.c to deparse patterns properly.


Current state:

The executor still uses regex matching, which works but has some
limitations.
Complex patterns with lots of alternations can be slow.

The 26-variable limit is still there since we're mapping to [a-z] for the
regex.

Reluctant quantifiers (+?, *?, ??) are parsed but will raise an error if you
try to use them. They'll need proper NFA support to work correctly.

Some optimization code was removed in the process, but it should work better
once NFA matching is in place.


Next steps:

The plan is to replace regex matching with NFA-based execution. The AST
structure is already set up for that.

The first step is converting the Thompson AST into a flat representation -
flattening pattern elements and building actual NFA states with transitions,
then replacing the regex engine with direct NFA matching. This should
handle all
the pattern syntax properly, including reluctant quantifiers.

Once that's working, the next step is handling multiple contexts during
matching -
tracking which states are active and absorbing context as we go through
rows.
This is essential for proper pattern matching behavior.

After that, I'll implement PATH and CLASSIFIER functions, which depend on
having
the match context available.

Then there's room for various optimizations - PATH space optimization,
incremental aggregate computation over matched rows, better state merging,
etc.

Longer-term, the goal is to get to MATCH_RECOGNIZE in the FROM clause for
full SQL standard compliance.

Let me know if you have any concerns or suggestions about the approach.

Best regards,
Henson

  146. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-12T10:45:07Z

    Hi Henson,

> Hi Ishii-san,
> 
> I've been working on the pattern grammar and AST for row pattern
> recognition.
> The attached patch extends the PATTERN clause to support more regex-like
> syntax.
> 
> The main additions are:
> 
> - Alternation: (A | B)
> - Grouping with quantifiers: (A B)+, (A | B)*
> - Full quantifier support: +, *, ?, {n}, {n,}, {n,m}
> - Reluctant quantifiers: +?, *?, ?? (parsed but not executed yet)

Excellent!

> I've introduced RPRPatternNode to represent the pattern as a Thompson-style
> AST with four node types: VAR, SEQ, ALT, and GROUP. Each node stores min/max
> bounds for quantifiers and a reluctant flag. This should make the eventual
> NFA implementation more straightforward.
> 
> The parser also does some basic pattern optimizations. For example:
>   - (A (B C)) gets flattened to (A B C)
>   - ((A)) unwraps to just A
>   - (A{2}){3} becomes A{6}
>   - (A | B | A) simplifies to (A | B)
>   - A A A merges into A{3,3}
> 
> I also updated ruleutils.c to deparse patterns properly.
> 
> 
> Current state:
> 
> The executor still uses regex matching, which works but has some
> limitations.
> Complex patterns with lots of alternations can be slow.
>
> The 26-variable limit is still there since we're mapping to [a-z] for the
> regex.
> 
> Reluctant quantifiers (+?, *?, ??) are parsed but will raise an error if you
> try to use them. They'll need proper NFA support to work correctly.
> 
> Some optimization code was removed in the process, but it should work better
> once NFA matching is in place.

Yeah. Now the rpr regression takes more than 7 seconds.  (with current
v37 patch it takes 86ms). I expect NFA matching works better.

> Next steps:
> 
> The plan is to replace regex matching with NFA-based execution. The AST
> structure is already set up for that.

Sounds promising.

> The first step is converting the Thompson AST into a flat representation -
> flattening pattern elements and building actual NFA states with transitions,
> then replacing the regex engine with direct NFA matching. This should
> handle all
> the pattern syntax properly, including reluctant quantifiers.
> 
> Once that's working, the next step is handling multiple contexts during
> matching -
> tracking which states are active and absorbing context as we go through
> rows.
> This is essential for proper pattern matching behavior.

Ok.

> After that, I'll implement PATH and CLASSIFIER functions, which depend on
> having
> the match context available.

What is PATH function? It's not in R010 or R020 as far as I know.

> Then there's room for various optimizations - PATH space optimization,
> incremental aggregate computation over matched rows, better state merging,
> etc.

Sounds good.

> Longer-term, the goal is to get to MATCH_RECOGNIZE in the FROM clause for
> full SQL standard compliance.

What about MEASURES clause? Without it, many things in the standard
cannot be implemented.

> Let me know if you have any concerns or suggestions about the approach.

Here are some comments on your patches.

- It is based on my v36 patches. But the latest one is v37 patch.  It
  would be nice if you create patches based on the my latest patches.

- You are doing some optimization (like (A (B C)) gets flattened to (A
  B C) etc.) in the parse analysis phase. I think doing that kind of
  optimization should be done in the optimizer/planner. Because with
  your patch a deparsed query (as shown by pg_get_viewdef()) looks
  different from what user inputs.

- If you add files to src/backend/parser, it should be noted in
  src/backend/parser/README.
  
- It would be noce to update typedefs patch if you add new typedefs.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  147. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-12T11:12:39Z

    >
> > After that, I'll implement PATH and CLASSIFIER functions, which depend on
> > having
> > the match context available.
>
> What is PATH function? It's not in R010 or R020 as far as I know.
>

A Match Path is essentially an array consisting of the CLASSIFIER() values
of all rows
within a matched sequence, arranged in chronological order.


> > Longer-term, the goal is to get to MATCH_RECOGNIZE in the FROM clause for
> > full SQL standard compliance.
>
> What about MEASURES clause? Without it, many things in the standard
> cannot be implemented.
>

I admit that my current understanding of the full SQL standard is still
evolving. However,
looking at the current codebase, I believed that the syntax I've
implemented so far
could be addressed even before mastering every detail of the standard.
That is why I have taken a 'code-first' approach for this stage.

Of course, reaching 'FULL' compliance is undoubtedly the ultimate goal.
Given
the complexity, I believe the features should be released in several
incremental phases.
I trust that you will help define the strategic roadmap for these releases.
As I continue to
deepen my understanding of the standard, I will do my best to contribute to
the
technical roadmap as well.



> > Let me know if you have any concerns or suggestions about the approach.
>
> Here are some comments on your patches.
>
> - It is based on my v36 patches. But the latest one is v37 patch.  It
>   would be nice if you create patches based on the my latest patches.
>

I will rebase my work on your latest v37 patch and move forward with that
for the next version. I’ll submit the updated patch soon.


> - You are doing some optimization (like (A (B C)) gets flattened to (A
>   B C) etc.) in the parse analysis phase. I think doing that kind of
>   optimization should be done in the optimizer/planner. Because with
>   your patch a deparsed query (as shown by pg_get_viewdef()) looks
>   different from what user inputs.
>

I completely agree that those optimizations belong in the optimizer/planner
phase.
I intentionally placed them in the parse analysis phase for now to make
debugging
easier during this initial development stage. Once the logic is stabilized,
I will move
them to the planner as you suggested to ensure proper deparsing.


> - If you add files to src/backend/parser, it should be noted in
>   src/backend/parser/README.
>
> - It would be noce to update typedefs patch if you add new typedefs.
>

Thank you for pointing those out. I had overlooked the README in
src/backend/parser,
and I will make sure to update it along with the typedefs patch in my next
submission.

Best regards,
Henson

  148. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-13T02:41:08Z

    >> What is PATH function? It's not in R010 or R020 as far as I know.
>>
> 
> A Match Path is essentially an array consisting of the CLASSIFIER() values
> of all rows
> within a matched sequence, arranged in chronological order.

So, the PATH function is Oracle specific? Or your own proposal?

>> > Longer-term, the goal is to get to MATCH_RECOGNIZE in the FROM clause for
>> > full SQL standard compliance.
>>
>> What about MEASURES clause? Without it, many things in the standard
>> cannot be implemented.
>>
> 
> I admit that my current understanding of the full SQL standard is still
> evolving. However,
> looking at the current codebase, I believed that the syntax I've
> implemented so far
> could be addressed even before mastering every detail of the standard.
> That is why I have taken a 'code-first' approach for this stage.
> 
> Of course, reaching 'FULL' compliance is undoubtedly the ultimate goal.
> Given
> the complexity, I believe the features should be released in several
> incremental phases.

I totally agree with the incremental approach.

> I trust that you will help define the strategic roadmap for these releases.
> As I continue to
> deepen my understanding of the standard, I will do my best to contribute to
> the
> technical roadmap as well.

Ok, here is my proposal for the strategic roadmap. This should be a
starting point of the discussion.

(1) The first release: this will a subset of R020 (RPR in Window
    clause), but will serve many of pratical uses of RPR.  The
    following features below will not be included.

    - MEASURES
    - SUBSET
    - SEEK
    - AFTER MATCH SKIP TO
    - AFTER MATCH SKIP TO FIRST
    - AFTER MATCH SKIP TO LAST
    - PERMUTE

Note 1: this list needs discussion and may be changed when the release
is made.

Note 2: Because MEASURES cannot be used in this release, all primary
row patter variables in the DEFINE clause must be unqualified ones
(i.e. "price", not "A.price") and they are implicitly qualified by the
universal row pattern variable. In this case the universal row pattern
variable is mapped to the current row, thus "price" in the DEFINE
clause is evaluated in the current row.

After (1) we can choose either (a) extend support for R020, i.e. add
more features to the first release, especially MEASURES or (b) support
a subset of R010 (MATCH_RECOGNIZE).

>> > Let me know if you have any concerns or suggestions about the approach.

> I will rebase my work on your latest v37 patch and move forward with that
> for the next version. I’ll submit the updated patch soon.

Thanks!

> I completely agree that those optimizations belong in the optimizer/planner
> phase.
> I intentionally placed them in the parse analysis phase for now to make
> debugging
> easier during this initial development stage. Once the logic is stabilized,
> I will move
> them to the planner as you suggested to ensure proper deparsing.

Understood. Looks like a good strategy.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  149. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-13T02:53:22Z

    >
> So, the PATH function is Oracle specific? Or your own proposal?
>


Match PATH is essentially the internal metadata or the trace log of which
Row Pattern Variable (Classifier) was assigned to each row during the
state machine's execution.

Match PATH is simply a set of Classifiers stored in the internal state.

> I trust that you will help define the strategic roadmap for these
> releases.
> > As I continue to
> > deepen my understanding of the standard, I will do my best to contribute
> to
> > the
> > technical roadmap as well.
>
> Ok, here is my proposal for the strategic roadmap. This should be a
> starting point of the discussion.
>
> (1) The first release: this will a subset of R020 (RPR in Window
>     clause), but will serve many of pratical uses of RPR.  The
>     following features below will not be included.
>
>     - MEASURES
>     - SUBSET
>     - SEEK
>     - AFTER MATCH SKIP TO
>     - AFTER MATCH SKIP TO FIRST
>     - AFTER MATCH SKIP TO LAST
>     - PERMUTE
>
> Note 1: this list needs discussion and may be changed when the release
> is made.
>
> Note 2: Because MEASURES cannot be used in this release, all primary
> row patter variables in the DEFINE clause must be unqualified ones
> (i.e. "price", not "A.price") and they are implicitly qualified by the
> universal row pattern variable. In this case the universal row pattern
> variable is mapped to the current row, thus "price" in the DEFINE
> clause is evaluated in the current row.
>
> After (1) we can choose either (a) extend support for R020, i.e. add
> more features to the first release, especially MEASURES or (b) support
> a subset of R010 (MATCH_RECOGNIZE).
>

I am currently fixing the code to satisfy the existing test cases.
Once the modifications are complete, I'll share the details so we can
discuss the differences between the current status and the proposed
roadmap.

Best regards,
Henson

  150. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-14T15:31:32Z

    Hi Ishii-san,


The attached patch implements streaming NFA-based pattern matching.
This is based on your v37 patch and the pattern grammar work I submitted
earlier.

This is a testable draft for review, not production-ready yet. It allows
in-depth verification of pattern optimization, pattern matching/merging,
context absorption, and window behavior.


Summary of changes:

1. NFA-based executor replacing regex approach
   - Streaming execution: processes one row at a time
   - Eliminates regex compilation overhead and 26-variable limit
   - Supports up to 252 pattern variables

   Note on 252 limit:
   Each match path stores a sequence of classifiers (one per matched row).
   Using 1-byte encoding keeps memory usage reasonable for long matches.
   Some values are reserved for pattern representation; this limit may
   decrease if more are needed. 200+ should be sufficient for practical use.

2. Pattern optimization moved to planner phase (as you suggested)
   - pg_get_viewdef() now shows original pattern
   - EXPLAIN shows optimized pattern
   - Optimizations: VAR merge (A A A -> A{3}), GROUP merge ((A B) (A B)+ ->
(A B){2,}),
     duplicate removal, SEQ/ALT flattening, quantifier multiplication
   - These optimizations also enable absorption for patterns that wouldn't
     otherwise qualify (e.g., (A B) (A B)+ -> (A B){2,} becomes absorbable)

3. Multi-context support with context absorption
   - Multiple concurrent NFA contexts for overlapping matches
   - Absorption optimization: earlier context absorbs later ones
     when they reach the same endpoint

   Note on absorption scope:
   I haven't fully worked out the theoretical bounds for safe absorption
yet.
   The current implementation covers most practically useful cases, but a
   complete solution would require more formal analysis.

   Current implementation handles cases where the behavior is clear:
   - Only patterns with unbounded first element (A+, A*, (A B)+, etc.)
   - Only when there's exactly one unbounded element at top level
   - Per-branch analysis for alternation patterns
   This may be extended as we better understand the theoretical limits.

   Examples:
   - A+ B     : absorbable (A+ unbounded first, same endpoint guaranteed)
   - A* B     : absorbable (A* unbounded first)
   - A B+     : NOT absorbable (unbounded not first, different start points)
   - A+ B+    : partial (absorbable while in A+, not after entering B+)
   - A? B     : NOT absorbable (? is bounded: {0,1})
   - A{2,} B  : absorbable (unbounded quantifier at first element)
   - A{2,5} B : NOT absorbable (bounded quantifier)
   - (A B){2,}: absorbable (GROUP with unbounded quantifier at top level)
   - A+ | B+  : absorbable per-branch (each branch analyzed independently)
   - A+ | B C : partial per-branch (A+ absorbable, B C not)
   - (A+ B)+   : NOT YET (absorbable at first A+ entry, nested analysis not
implemented)

4. Reluctant quantifiers
   - Will be supported in a follow-up patch
   - I need to study some edge cases in the spec first

Also updated: parser/README, typedefs.list


Testing:

The rpr regression test has been expanded significantly with tests for:
- Alternation and grouping patterns
- Quantifiers ({n}, {n,}, {n,m}, +, *, ?)
- Pattern optimization verification
- Multi-context matching
- Context absorption behavior
- Lexical order compliance

Below is a reference query designed to test NFA functionality in a clean,
unambiguous way. Using explicit ARRAY flags eliminates edge cases and makes
the pattern matching behavior completely predictable, allowing clear
verification
of multi-context execution and alternation handling.

Example (multi-context overlapping match test):
Uses ARRAY flags with ANY() to make each row's pattern membership explicit,
making test behavior predictable and easy to verify.

  WITH data AS (
    SELECT * FROM (VALUES
      (1, ARRAY['A']), (2, ARRAY['B']), (3, ARRAY['C']),
      (4, ARRAY['D']), (5, ARRAY['E']), (6, ARRAY['F'])
    ) AS t(id, flags)
  )
  SELECT id, flags, first_value(id) OVER w AS start, last_value(id) OVER w
AS end
  FROM data
  WINDOW w AS (
    ORDER BY id
    ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
    AFTER MATCH SKIP TO NEXT ROW
    PATTERN (A B C D E | B C D | C D E F)
    DEFINE A AS 'A' = ANY(flags), B AS 'B' = ANY(flags), ...
  );

  -- Result (SKIP TO NEXT ROW enables multi-context):
   id | flags | start | end
  ----+-------+-------+-----
    1 | {A}   |     1 |   5   <- A B C D E
    2 | {B}   |     2 |   4   <- B C D
    3 | {C}   |     3 |   6   <- C D E F
    4 | {D}   |       |
    5 | {E}   |       |
    6 | {F}   |       |


Performance:

Regression test timing (rpr test):

  Parallel group (with other tests):
    Before (v37 regex):  198 ms
    After (NFA):         123 ms  (~38% faster)
    After (NFA + tests): 127 ms  (still ~36% faster)

  Single test (isolated):
    Before (v37 regex):  38 ms
    After (NFA):         26 ms  (~32% faster)
    After (NFA + tests): 41 ms  (+8%, more tests added)

Note: The rpr test was moved out of the parallel group in the schedule
for more accurate timing measurement.

The streaming NFA approach eliminates regex compilation overhead per-row
and reduces redundant DEFINE evaluations, resulting in better performance
especially for complex patterns.


Let me know if you have any questions or concerns.


Best regards,
Henson

>

  151. Re: Row pattern recognition

    Jacob Champion <jacob.champion@enterprisedb.com> — 2026-01-14T18:12:07Z

    On Sat, Jan 10, 2026 at 1:20 AM Henson Choi <assam258@gmail.com> wrote:
> I'd definitely be interested in seeing it, rough edges and all. I haven't
> tackled the PostgreSQL integration design yet, so understanding how
> preferment rules map to the existing infrastructure would be very helpful.
> (Tatsuo found that email, by the way.)

Yep -- the dev branch link from that email still works, for you and
anyone interested:

    https://github.com/jchampio/postgres/tree/dev/rpr

--Jacob

  152. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-15T03:46:16Z

    Hi Jacob,

Thank you for sharing your dev branch! I extracted test cases from your
implementation and found two failing patterns in my executor:

2026년 1월 15일 (목) AM 3:12, Jacob Champion <jacob.champion@enterprisedb.com>님이
작성:

> On Sat, Jan 10, 2026 at 1:20 AM Henson Choi <assam258@gmail.com> wrote:
> > I'd definitely be interested in seeing it, rough edges and all. I haven't
> > tackled the PostgreSQL integration design yet, so understanding how
> > preferment rules map to the existing infrastructure would be very
> helpful.
> > (Tatsuo found that email, by the way.)
>
> Yep -- the dev branch link from that email still works, for you and
> anyone interested:
>
>     https://github.com/jchampio/postgres/tree/dev/rpr
>

1. (A | B)+ C with data A, B, A, C

My result: NULL (no match)
Correct: 1-4

2. ((A | B) C)+ with data A, C, B, C, X

My result: 1-2, 3-4 (split matches)
Correct: 1-4


> --Jacob
>

Since I implemented the pattern matching and context absorption somewhat
naively,
I think these areas will need an overall review. Thanks for the helpful
test patterns!

Best regards,
Henson

  153. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-15T04:56:24Z

    Hi Henson,

> Hi Ishii-san,
> 
> 
> The attached patch implements streaming NFA-based pattern matching.
> This is based on your v37 patch and the pattern grammar work I submitted
> earlier.

Great!

> This is a testable draft for review, not production-ready yet. It allows
> in-depth verification of pattern optimization, pattern matching/merging,
> context absorption, and window behavior.

I squashed your patches into my v37 patches and created v38 patches.
(See attached). Will it be Okay for you to work on the patches from
now on?

Also I have added you to the commit message as an author.

> Summary of changes:

Will check.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  154. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-15T08:25:52Z

    Hi Henson,

The merged v38 patch passed CI check:
https://cirrus-ci.com/github/postgresql-cfbot/postgresql/cf%2F4460

Congratulations!

>> Summary of changes:
> 
> Will check.

I have one question regaring your patch.

In update_reduced_frame():
+	hasLimitedFrame = (frameOptions & FRAMEOPTION_ROWS) &&
+					  !(frameOptions & FRAMEOPTION_END_UNBOUNDED_FOLLOWING);
+	if (hasLimitedFrame && winstate->endOffsetValue != 0)
+		frameOffset = DatumGetInt64(winstate->endOffsetValue);

frameOffset is a offset value n from "ROWS BETWEEN CURRENT ROW AND n FOLLOWING".
Later you use this here:

+	ctxFrameEnd = ctx->matchStartRow + frameOffset + 1;

So my question is, how do you ensure that ctxFrameEnd does not go
beyond the full window frame end?

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  155. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-15T09:31:20Z

    Hi Ishii-san,

I have one question regaring your patch.
>
> In update_reduced_frame():
> +       hasLimitedFrame = (frameOptions & FRAMEOPTION_ROWS) &&
> +                                         !(frameOptions &
> FRAMEOPTION_END_UNBOUNDED_FOLLOWING);
> +       if (hasLimitedFrame && winstate->endOffsetValue != 0)
> +               frameOffset = DatumGetInt64(winstate->endOffsetValue);
>
> frameOffset is a offset value n from "ROWS BETWEEN CURRENT ROW AND n
> FOLLOWING".
> Later you use this here:
>
> +       ctxFrameEnd = ctx->matchStartRow + frameOffset + 1;
>
> So my question is, how do you ensure that ctxFrameEnd does not go
> beyond the full window frame end?
>

In update_reduced_frame():

    frameOffset = endOffsetValue;     // e.g., 2 from "2 FOLLOWING"

    for each row (currentPos):

        if (!rowExists)               // partition end reached
            finalize all contexts;
            break;

        for each context:
            ctxFrameEnd = matchStartRow + frameOffset + 1;
            if (currentPos >= ctxFrameEnd)
                finalize this context;
                continue;

Even if ctxFrameEnd exceeds partition end, the "if (!rowExists)" check
fires first and finalizes all contexts at the actual partition boundary.

Best regards,
Henson

  156. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-15T12:04:57Z

    Hi Henson,

>> So my question is, how do you ensure that ctxFrameEnd does not go
>> beyond the full window frame end?
>>
> 
> In update_reduced_frame():
> 
>     frameOffset = endOffsetValue;     // e.g., 2 from "2 FOLLOWING"
> 
>     for each row (currentPos):
> 
>         if (!rowExists)               // partition end reached
>             finalize all contexts;
>             break;
> 
>         for each context:
>             ctxFrameEnd = matchStartRow + frameOffset + 1;
>             if (currentPos >= ctxFrameEnd)
>                 finalize this context;
>                 continue;
> 
> Even if ctxFrameEnd exceeds partition end, the "if (!rowExists)" check
> fires first and finalizes all contexts at the actual partition boundary.

We need to check the *frame" end, not the partition end.

I think your patch relies on !window_gettupleslot() to check whether
the row exists.

	if (!window_gettupleslot(winobj, pos, slot))
		return false;	/* No row exists */

But the function only checks the row existence in the current partition:

 *	Fetch the pos'th tuple of the current partition into the slot,

Thus it is possible that window_gettupleslot() returns true but the
row is not in the current frame in case that the partition is divided
into some frames. You need to check the row existence in a frame. For
this purpose you can use row_is_in_frame().

I ran following query and got the result with v38 (which includes your
NFA patches).

WITH data AS (
 SELECT * FROM (VALUES
  ('A', 1), ('A', 2),
  ('B', 3), ('B', 4)
  ) AS t(gid, id))
  SELECT gid, id, array_agg(id) OVER w
  FROM data
  WINDOW w AS (
   PARTITION BY gid
   ROWS BETWEEN CURRENT ROW AND 2 FOLLOWING
   AFTER MATCH SKIP TO NEXT ROW
   PATTERN (A+)
   DEFINE A AS id < 10
  );
 gid | id | array_agg 
-----+----+-----------
 A   |  1 | {1,2}
 A   |  2 | 
 B   |  3 | {3,4}
 B   |  4 | 
(4 rows)

I think the second and 4th rows are expected to return some data in
array_agg colum. In fact v37 patch returns following results for the
same query:

 gid | id | array_agg 
-----+----+-----------
 A   |  1 | {1,2}
 A   |  2 | {2}
 B   |  3 | {3,4}
 B   |  4 | {4}
(4 rows)

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  157. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-15T13:28:39Z

    Hi Ishii-san

We need to check the *frame" end, not the partition end.
>
> I think your patch relies on !window_gettupleslot() to check whether
> the row exists.
>
>         if (!window_gettupleslot(winobj, pos, slot))
>                 return false;   /* No row exists */
>
> But the function only checks the row existence in the current partition:
>
>  *      Fetch the pos'th tuple of the current partition into the slot,
>
> Thus it is possible that window_gettupleslot() returns true but the
> row is not in the current frame in case that the partition is divided
> into some frames. You need to check the row existence in a frame. For
> this purpose you can use row_is_in_frame().
>
> I ran following query and got the result with v38 (which includes your
> NFA patches).
>
> WITH data AS (
>  SELECT * FROM (VALUES
>   ('A', 1), ('A', 2),
>   ('B', 3), ('B', 4)
>   ) AS t(gid, id))
>   SELECT gid, id, array_agg(id) OVER w
>   FROM data
>   WINDOW w AS (
>    PARTITION BY gid
>    ROWS BETWEEN CURRENT ROW AND 2 FOLLOWING
>    AFTER MATCH SKIP TO NEXT ROW
>    PATTERN (A+)
>    DEFINE A AS id < 10
>   );
>  gid | id | array_agg
> -----+----+-----------
>  A   |  1 | {1,2}
>  A   |  2 |
>  B   |  3 | {3,4}
>  B   |  4 |
> (4 rows)
>
> I think the second and 4th rows are expected to return some data in
> array_agg colum. In fact v37 patch returns following results for the
> same query:
>
>  gid | id | array_agg
> -----+----+-----------
>  A   |  1 | {1,2}
>  A   |  2 | {2}
>  B   |  3 | {3,4}
>  B   |  4 | {4}
> (4 rows)
>

While investigating the issue you mentioned, I found that the problem
is caused by the context absorption logic, not the frame boundary check.

The absorption logic removes a newer context when
older->matchEndRow >= newer->matchEndRow:

Example with pattern A+:
- Context at row 1: matchEndRow = 2 (match {1,2})
- Context at row 2: matchEndRow = 2 (match {2})
→ Newer context absorbed, row 2's result lost

With SKIP TO NEXT ROW, each row should produce its own independent match.
The absorption was incorrectly removing these valid overlapping matches.

I modified the absorption logic to only apply to SKIP PAST LAST ROW mode,
where it provides performance benefits without affecting correctness.

For SKIP TO NEXT ROW, absorption is now disabled, allowing each row
to produce independent overlapping matches as expected:

 gid | id | array_agg
-----+----+-----------
 A   |  1 | {1,2}
 A   |  2 | {2}
 B   |  3 | {3,4}
 B   |  4 | {4}
(4 rows)

I'm attaching 3 patches:

1. Test cases from Jacob Champion's branch
   - Added 355 lines of test cases and 51 lines of executor changes

2. Refactored update_reduced_frame for readability and performance
   - Extracted large code blocks into separate functions
   - Early return for already-processed positions
   - Integrated nfa_unlink_context into nfa_context_free
   - Used oldest-first context ordering for early termination

3. Enable context absorption only for SKIP PAST LAST ROW
   - Fixes overlapping match issue for SKIP TO NEXT ROW
   - In my 5000-row test, absorption showed significant performance gain
for SKIP PAST LAST ROW
   - Added your test case to regression tests

Since I implemented it in the order of parser/planner/nfa/executor,
the executor flow became somewhat confusing.
I plan to continue improving it by reviewing and refactoring
from the top-level functions downward.

During refactoring, I plan to simplify pattern optimization and context
absorption logic, keeping only the parts that are clearly correct, even
if this means some performance loss in the initial version. We can add
optimizations back later once the core logic is stable and well-tested.

Best regards,
Henson

  158. Re: Row pattern recognition

    Jacob Champion <jacob.champion@enterprisedb.com> — 2026-01-15T16:06:53Z

    On Wed, Jan 14, 2026 at 7:46 PM Henson Choi <assam258@gmail.com> wrote:
> Since I implemented the pattern matching and context absorption somewhat naively,
> I think these areas will need an overall review. Thanks for the helpful test patterns!

You're welcome, glad it was useful!

--Jacob

  159. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-15T23:44:42Z

    Hi Ishii-san,

Thank you for raising the frame boundary concern. You're right that
different contexts have different frame boundaries.

2026년 1월 15일 (목) PM 10:28, Henson Choi <assam258@gmail.com>님이 작성:

> Hi Ishii-san
>
> We need to check the *frame" end, not the partition end.
>>
>> I think your patch relies on !window_gettupleslot() to check whether
>> the row exists.
>>
>>         if (!window_gettupleslot(winobj, pos, slot))
>>                 return false;   /* No row exists */
>>
>> But the function only checks the row existence in the current partition:
>>
>>  *      Fetch the pos'th tuple of the current partition into the slot,
>>
>> Thus it is possible that window_gettupleslot() returns true but the
>> row is not in the current frame in case that the partition is divided
>> into some frames. You need to check the row existence in a frame. For
>> this purpose you can use row_is_in_frame().
>>
>
Your suggestion was to use row_is_in_frame() to check frame boundaries
for each row. However, I took a simpler approach: just disable context
absorption entirely when the frame is limited.

The root cause is that context absorption assumes all contexts see
the same rows. With limited frames, each context starting at a different
row has a different visible range, so we cannot absorb one context into
another.

As I mentioned before, I think we should use absorption conservatively
for now - only in cases where it's clearly safe (e.g., A* at the
beginning of the pattern). We can extend it later through more in-depth
research.

The fix:

    if (winstate->rpSkipTo == ST_PAST_LAST_ROW && !hasLimitedFrame)
        nfa_absorb_contexts(winstate, targetCtx, currentPos);

I added a test case to verify this:

    -- Pattern: A+ B, Data: A(0) A(1) A(2) B(3), Frame: CURRENT ROW AND 2
FOLLOWING
    -- Row 0: frame [0,2], cannot see B at row 3 -> no match
    -- Row 1: frame [1,3], can see A A B -> should match rows 1-3
    WITH frame_absorb_test AS (
     SELECT * FROM (VALUES
      (0, 'A'), (1, 'A'), (2, 'A'), (3, 'B')
     ) AS t(id, flag)
    )
    SELECT id, flag, first_value(id) OVER w AS match_start, last_value(id)
OVER w AS match_end
    FROM frame_absorb_test
    WINDOW w AS (
     ORDER BY id
     ROWS BETWEEN CURRENT ROW AND 2 FOLLOWING
     AFTER MATCH SKIP PAST LAST ROW
     PATTERN (A+ B)
     DEFINE
      A AS flag = 'A',
      B AS flag = 'B'
    );

Before fix (absorption incorrectly absorbs row 1's context into row 0's):

     id | flag | match_start | match_end
    ----+------+-------------+-----------
      0 | A    |             |
      1 | A    |             |
      2 | A    |             |
      3 | B    |             |

After fix (row 1 correctly matches):

     id | flag | match_start | match_end
    ----+------+-------------+-----------
      0 | A    |             |
      1 | A    |           1 |         3
      2 | A    |             |
      3 | B    |             |

Note that for SKIP TO NEXT ROW, absorption is already disabled,
so your test case with SKIP TO NEXT ROW should
work correctly.

I'm attaching three related commits:

1. Row pattern recognition: Improve NFA state memory management

   This commit refactors NFA state management for better readability
   and maintainability.

2. Row pattern recognition: Disable context absorption for limited frame

   This commit further restricts absorption: even with SKIP PAST LAST ROW,
   absorption is disabled when the frame is limited. With limited frames,
   different contexts have different frame boundaries, making absorption
   unsafe.


>
>> I ran following query and got the result with v38 (which includes your
>> NFA patches).
>>
>> WITH data AS (
>>  SELECT * FROM (VALUES
>>   ('A', 1), ('A', 2),
>>   ('B', 3), ('B', 4)
>>   ) AS t(gid, id))
>>   SELECT gid, id, array_agg(id) OVER w
>>   FROM data
>>   WINDOW w AS (
>>    PARTITION BY gid
>>    ROWS BETWEEN CURRENT ROW AND 2 FOLLOWING
>>    AFTER MATCH SKIP TO NEXT ROW
>>    PATTERN (A+)
>>    DEFINE A AS id < 10
>>   );
>>  gid | id | array_agg
>> -----+----+-----------
>>  A   |  1 | {1,2}
>>  A   |  2 |
>>  B   |  3 | {3,4}
>>  B   |  4 |
>> (4 rows)
>>
>> I think the second and 4th rows are expected to return some data in
>> array_agg colum. In fact v37 patch returns following results for the
>> same query:
>>
>>  gid | id | array_agg
>> -----+----+-----------
>>  A   |  1 | {1,2}
>>  A   |  2 | {2}
>>  B   |  3 | {3,4}
>>  B   |  4 | {4}
>> (4 rows)
>>
>
> While investigating the issue you mentioned, I found that the problem
> is caused by the context absorption logic, not the frame boundary check.
>
> The absorption logic removes a newer context when
> older->matchEndRow >= newer->matchEndRow:
>
> Example with pattern A+:
> - Context at row 1: matchEndRow = 2 (match {1,2})
> - Context at row 2: matchEndRow = 2 (match {2})
> → Newer context absorbed, row 2's result lost
>
> With SKIP TO NEXT ROW, each row should produce its own independent match.
> The absorption was incorrectly removing these valid overlapping matches.
>
> I modified the absorption logic to only apply to SKIP PAST LAST ROW mode,
> where it provides performance benefits without affecting correctness.
>
> For SKIP TO NEXT ROW, absorption is now disabled, allowing each row
> to produce independent overlapping matches as expected:
>
>  gid | id | array_agg
> -----+----+-----------
>  A   |  1 | {1,2}
>  A   |  2 | {2}
>  B   |  3 | {3,4}
>  B   |  4 | {4}
> (4 rows)
>
> I'm attaching 3 patches:
>
> 1. Test cases from Jacob Champion's branch
>    - Added 355 lines of test cases and 51 lines of executor changes
>
> 2. Refactored update_reduced_frame for readability and performance
>    - Extracted large code blocks into separate functions
>    - Early return for already-processed positions
>    - Integrated nfa_unlink_context into nfa_context_free
>    - Used oldest-first context ordering for early termination
>
> 3. Enable context absorption only for SKIP PAST LAST ROW
>    - Fixes overlapping match issue for SKIP TO NEXT ROW
>    - In my 5000-row test, absorption showed significant performance gain
> for SKIP PAST LAST ROW
>    - Added your test case to regression tests
>
> Since I implemented it in the order of parser/planner/nfa/executor,
> the executor flow became somewhat confusing.
> I plan to continue improving it by reviewing and refactoring
> from the top-level functions downward.
>
> During refactoring, I plan to simplify pattern optimization and context
> absorption logic, keeping only the parts that are clearly correct, even
> if this means some performance loss in the initial version. We can add
> optimizations back later once the core logic is stable and well-tested.
>

I'm now reviewing the remaining parts: pattern optimization in planner,
absorption function, and step function. Once the review is complete,
I'll send the rest of the patches.

Best regards,
Henson

  160. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-16T04:44:22Z

    Hi Henson,

> Hi Ishii-san,
> 
> Thank you for raising the frame boundary concern. You're right that
> different contexts have different frame boundaries.

Ok.

> Your suggestion was to use row_is_in_frame() to check frame boundaries
> for each row. However, I took a simpler approach: just disable context
> absorption entirely when the frame is limited.
> 
> The root cause is that context absorption assumes all contexts see
> the same rows. With limited frames, each context starting at a different
> row has a different visible range, so we cannot absorb one context into
> another.
> 
> As I mentioned before, I think we should use absorption conservatively
> for now - only in cases where it's clearly safe (e.g., A* at the
> beginning of the pattern). We can extend it later through more in-depth
> research.

Agreed.

> The fix:
> 
>     if (winstate->rpSkipTo == ST_PAST_LAST_ROW && !hasLimitedFrame)
>         nfa_absorb_contexts(winstate, targetCtx, currentPos);
> 
> I added a test case to verify this:
[snip]
> I'm attaching three related commits:
> 
> 1. Row pattern recognition: Improve NFA state memory management
> 
>    This commit refactors NFA state management for better readability
>    and maintainability.
> 
> 2. Row pattern recognition: Disable context absorption for limited frame
> 
>    This commit further restricts absorption: even with SKIP PAST LAST ROW,
>    absorption is disabled when the frame is limited. With limited frames,
>    different contexts have different frame boundaries, making absorption
>    unsafe.

I have applied the patches on top of v38 to create v39 patches.
In addition to these patches, I have made following changes:

- Add check for frame option "EXCLUDE". With RPR, using EXCLUDE is not
  permitted by the SQL standard. A test case added.

- Remove unused typedef from windowfuncs.c (which causes removal of the
  typedef from typedes.list). The typedef was accidentally left by me
  while developing RPR.

- Update the Copyright year to 2026 in some newly added files.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  161. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-17T08:18:50Z

    Hi Henson,

> I'm now reviewing the remaining parts: pattern optimization in planner,
> absorption function, and step function. Once the review is complete,
> I'll send the rest of the patches.

I am reviewing the test cases and found this:

SELECT * FROM (SELECT 1 AS x) t
 WINDOW w AS (
 ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
 PATTERN (a b c d e f g h i j k l m n o p q r s t u v w x y z aa)
 DEFINE a AS TRUE
);
ERROR:  number of row pattern definition variable names exceeds 26

I thought you have eliminated the 26 variables limit. If so, related
codes using WindowClause->defineInitial should be removed. Shall I do
that? Or you want to do that while refactoring?

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  162. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-17T08:33:28Z

    2026년 1월 17일 (토) 17:19, Tatsuo Ishii <ishii@postgresql.org>님이 작성:

> Hi Henson,
>
> > I'm now reviewing the remaining parts: pattern optimization in planner,
> > absorption function, and step function. Once the review is complete,
> > I'll send the rest of the patches.
>
> I am reviewing the test cases and found this:
>
> SELECT * FROM (SELECT 1 AS x) t
>  WINDOW w AS (
>  ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>  PATTERN (a b c d e f g h i j k l m n o p q r s t u v w x y z aa)
>  DEFINE a AS TRUE
> );
> ERROR:  number of row pattern definition variable names exceeds 26
>
> I thought you have eliminated the 26 variables limit. If so, related
> codes using WindowClause->defineInitial should be removed. Shall I do
> that? Or you want to do that while refactoring?
>


Already found and cleaned up.
Will include it in the next patch.​​​​​​​​​​​​​​​​


> Best regards,
> --
> Tatsuo Ishii
> SRA OSS K.K.
> English: http://www.sraoss.co.jp/index_en/
> Japanese:http://www.sraoss.co.jp
>

  163. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-17T08:38:43Z

    Hi Henson,

> Already found and cleaned up.
> Will include it in the next patch.​​​​​​​​​​​​​​​​

Excellent! Thank you so much.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  164. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-17T12:37:38Z

    Hi Ishii-san,

2026년 1월 17일 (토) PM 5:38, Tatsuo Ishii <ishii@postgresql.org>님이 작성:

> Hi Henson,
>
> > Already found and cleaned up.
> > Will include it in the next patch.
>
> Excellent! Thank you so much.
>

 Attached is the patch with the following changes:

1. Removed 26 variables limit and fixed count overflow:
   - Deleted defineInitial field from WindowClause (parsenodes.h),
     WindowAgg (plannodes.h), and related code in createplan.c and
parse_rpr.c
   - Now uses RPR_VARID_MAX (252) constant for the limit check
   - Fixed NFA count overflow issue:
     * Changed RPRNFAState.counts from int16 to int32 (execnodes.h)
     * Added RPR_COUNT_MAX (INT32_MAX) constant (rpr.h)
     * Handles pattern matching failures correctly at >32,767 repetitions
     * Added INT32_MAX boundary handling to prevent overflow
   - Updated test cases in rpr.sql:
     * Updated variable limit tests from 26 to 252
     * Added 100K row large partition test to verify int32 overflow handling

2. Simplified context absorption logic (under review):
   - Key insight: Most absorption opportunities provide no practical benefit
     because subsequent contexts are discarded immediately after a match
     succeeds. Only absorption at unbounded pattern start shows measurable
impact.
   - Changes:
     * Planner (rpr.c): Added isUnboundedStart() to identify patterns where
       leading unbounded quantifiers enable safe absorption
     * Executor (nodeWindowAgg.c): Simplified to only absorb at pattern
start,
       separated cleanup logic into nfa_cleanup_dead_contexts()
     * Added pattern->isAbsorbable flag to enable absorption only when safe
   - Impact: Reduces context growth from quadratic to linear.
     Example from rpr_explain.sql Test 3.1 (PATTERN A+ B, 50 rows):
     * With absorption: 51 total contexts, 3 peak, 30 absorbed
     * Without absorption: Would create O(n²) contexts (each row starts a
       context, creating 1+2+3+4 contexts per 5-row segment)
     * Absorption maintains linear O(n) growth instead of quadratic
explosion
   - Note: Changes in EXPLAIN ANALYZE output suggest there may be issues in
     the matching process, particularly in absorption and NFA state
transitions.
     I need to review these areas more thoroughly before I can be confident
in
     their correctness.

3. Added NFA statistics to EXPLAIN ANALYZE output (under review):
   - Extended explain.c to show NFA construction and matching statistics
   - Added comprehensive statistics tracking (execnodes.h):
     * NFALengthStats structure for min/max/avg length measurements
     * State statistics: peak, total created, merged (deduplicated)
     * Context statistics: peak, total created, absorbed, skipped, pruned
     * Match statistics: succeeded/failed counts with length distributions
   - Implemented statistics collection in nodeWindowAgg.c
   - New test suite: rpr_explain.sql for testing NFA statistics output
   - Note: rpr_explain is not included in parallel_schedule (runs
separately)
   - Statistics reveal pattern complexity impact:
     Example from Test 2.7 (PATTERN ((A | B)+)+, 1000 rows):
     * NFA States: 2,664 peak, 892,447 total created
     * Peak grows linearly with rows (~2.6x), likely due to state merging
     * High total shows significant state churn despite controlled peak
     * This demonstrates complex pattern behavior that users should
understand
     * Question: Should we add a GUC parameter to limit NFA state peak count
       (e.g., max_rpr_nfa_states) to provide administrators with safety
controls?
   - This feature is still under review and may be refined in future patches

4. Major refactoring of pattern compilation and optimization (rpr.c):
   - Split large monolithic functions into smaller, semantically meaningful
     single-purpose functions for better readability
   - Pattern compilation refactored:
     * Added collectDefineVariables() to populate variable names from DEFINE
     * Split scanRPRPattern() into recursive worker and wrapper
     * Extracted allocateRPRPattern() for memory allocation
     * Split fillRPRPattern() by node type: fillRPRPatternVar(),
       fillRPRPatternGroup(), fillRPRPatternAlt()
   - Pattern optimization converted to dispatcher pattern:
     * optimizeSeqPattern() - flattens nested SEQ, merges consecutive
vars/groups
     * optimizeAltPattern() - flattens nested ALT, removes duplicates
     * optimizeGroupPattern() - multiplies quantifiers, unwraps single
children
   - Each optimization step broken into separate helper functions:
     * flattenSeqChildren(), mergeConsecutiveVars(),
mergeConsecutiveGroups()
     * flattenAltChildren(), removeDuplicateAlternatives()
     * tryMultiplyQuantifiers(), tryUnwrapGroup()
   - New optimization: mergeGroupPrefixSuffix()
     * A B (A B)+ -> (A B){2,}
     * (A B)+ A B -> (A B){2,}
     * A B (A B)+ A B -> (A B){3,}
   - Code structure reorganized with categorized forward declarations
   - Added test cases in rpr.sql to verify new optimizations:
     * Consecutive GROUP merge tests
     * Quantifier multiplication tests (including INF blocking)
     * Group prefix/suffix merge tests

5. Fixed quantifier validation:
   - Corrected bounds checking in gram.y (e.g., {,m} now requires m >= 1)
   - Improved error messages for invalid quantifier ranges

Additional notes:
- The NFA statistics output format in EXPLAIN ANALYZE was quickly
  implemented for debugging purposes. I would appreciate feedback from
  experienced PostgreSQL developers on:
  * Whether the output format and structure are appropriate
  * Whether these statistics would be valuable for query tuning in
production,
    or if they are only useful for development/debugging
  * If production-worthy, what simplifications might make them more
accessible
    to users unfamiliar with NFA internals
- Reluctant quantifier handling needs further clarification. If I cannot
  make confident fixes before the next review cycle, I may temporarily
  raise errors for reluctant cases until the logic is properly sorted out.

Next steps:
- The executor implementation (nodeWindowAgg.c) is the only major component
  remaining for detailed review.
- I will focus on thoroughly reviewing the absorption logic and NFA state
  transition mechanisms to address the concerns noted in section 2.
- After making necessary improvements, I will send the next patch with
  more confidence in the executor correctness.

Best regards,
Henson

  165. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-18T07:52:43Z

    Hi Henson,

>  Attached is the patch with the following changes:
[snip]

After applying your patch on top of v39 patch sets, I see compiler
warnings:

nodeWindowAgg.c: In function ‘nfa_absorb_contexts’:
nodeWindowAgg.c:5565:47: warning: the comparison will always evaluate as ‘true’ for the address of ‘counts’ will never be NULL [-Waddress]
 5565 |                                 if (s->counts != NULL && s->counts[0] > newerMaxCount)
      |                                               ^~
In file included from ../../../src/include/executor/execdesc.h:18,
                 from ../../../src/include/executor/executor.h:18,
                 from nodeWindowAgg.c:41:
../../../src/include/nodes/execnodes.h:2532:25: note: ‘counts’ declared here
 2532 |         int32           counts[FLEXIBLE_ARRAY_MEMBER];  /* repetition counts by depth */
      |                         ^~~~~~
nodeWindowAgg.c:5572:47: warning: the comparison will always evaluate as ‘true’ for the address of ‘counts’ will never be NULL [-Waddress]
 5572 |                                 if (s->counts != NULL && s->counts[0] > olderMaxCount)
      |                                               ^~
../../../src/include/nodes/execnodes.h:2532:25: note: ‘counts’ declared here
 2532 |         int32           counts[FLEXIBLE_ARRAY_MEMBER];  /* repetition counts by depth */
      |                         ^~~~~~
nodeWindowAgg.c:5429:33: warning: variable ‘maxDepth’ set but not used [-Wunused-but-set-variable]
 5429 |         int                     maxDepth;
      |                                 ^~~~~~~~

Shall I fix them (and post v40 patches), or would you like to fix and
post another patch?

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  166. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-18T08:45:06Z

    >
> Shall I fix them (and post v40 patches), or would you like to fix and
> post another patch?
>

Yes, please fix them and post v40. Thanks!

  167. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-18T09:26:15Z

    >> Shall I fix them (and post v40 patches), or would you like to fix and
>> post another patch?
>>
> 
> Yes, please fix them and post v40. Thanks!

Got it.

BTW, I found IGNORE NULLS module (ignorenulls_getfuncarginframe) needs
to be fixed. The module is used when a window function is called with
IGNORE NULLS option. Currently PostgreSQL ignores the option if RPR is
used because ignorenulls_getfuncarginframe does not call
row_is_in_reduced_frame(). I will bring in the fix into v40.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  168. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-18T11:49:02Z

    Hi Hackers,

I’m writing to leave a brief note on State Pruning, a potential
optimization for the RPR executor.

To be clear, this is a separate concern from both context absorption and
state merging. While state merging handles identical states within a
context, and context absorption handles dominance between different
contexts, State Pruning focuses on discarding active NFA states that have
become irrelevant due to lexical priority rules.

The core idea is to prune individual states by comparing them with the
matchedState already secured. If a high-priority branch has already
recorded a match, any active states in lower-priority branches may become
"dead wood"—even if they are currently matching, they can never supersede
the existing winner.

Detailed Example: A+ (prio 0) | B+ (prio 1) Consider an input where both A
and B are TRUE for consecutive rows:

Row 1 (A=TRUE, B=TRUE):

A+ matches and records a matchedState (length 1, priority 0).

B+ also matches (length 1, priority 1) and remains in ctx->states.

Pruning Point: Since a priority 0 match is already secured, B+ (priority 1)
can never win, regardless of how much longer it matches. We can prune the
B+ state here.

Row 2 (A=TRUE, B=TRUE):

Without pruning: The engine evaluates both A+ and B+.

With pruning: Only A+ is evaluated. matchedState is updated to length 2.

Row 3 (A=FALSE, B=FALSE):

A+ fails. The final result is the matchedState from Row 2 (A+ with length
2).

As shown, State Pruning can significantly reduce the NFA workload by
stopping redundant branches early. However, implementing this correctly
will require substantial research and careful analysis, especially when
dealing with the complex interplay between greedy/reluctant quantifiers and
branch convergence. We must ensure that pruning a state won't lead to
incorrect results in more complex scenarios.

I am documenting this as a future research item. For now, my focus remains
on stabilizing the core RPR implementation and context absorption.

Regards,
Henson

2026년 1월 18일 (일) PM 6:26, Tatsuo Ishii <ishii@postgresql.org>님이 작성:

> >> Shall I fix them (and post v40 patches), or would you like to fix and
> >> post another patch?
> >>
> >
> > Yes, please fix them and post v40. Thanks!
>
> Got it.
>
> BTW, I found IGNORE NULLS module (ignorenulls_getfuncarginframe) needs
> to be fixed. The module is used when a window function is called with
> IGNORE NULLS option. Currently PostgreSQL ignores the option if RPR is
> used because ignorenulls_getfuncarginframe does not call
> row_is_in_reduced_frame(). I will bring in the fix into v40.
>
> Best regards,
> --
> Tatsuo Ishii
> SRA OSS K.K.
> English: http://www.sraoss.co.jp/index_en/
> Japanese:http://www.sraoss.co.jp
>

  169. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-18T12:09:29Z

    Hi Henson,

>  Attached is the patch with the following changes:
> 
> 1. Removed 26 variables limit and fixed count overflow:
>    - Deleted defineInitial field from WindowClause (parsenodes.h),
>      WindowAgg (plannodes.h), and related code in createplan.c and
> parse_rpr.c
>    - Now uses RPR_VARID_MAX (252) constant for the limit check
>    - Fixed NFA count overflow issue:
>      * Changed RPRNFAState.counts from int16 to int32 (execnodes.h)
>      * Added RPR_COUNT_MAX (INT32_MAX) constant (rpr.h)
>      * Handles pattern matching failures correctly at >32,767 repetitions
>      * Added INT32_MAX boundary handling to prevent overflow
>    - Updated test cases in rpr.sql:
>      * Updated variable limit tests from 26 to 252
>      * Added 100K row large partition test to verify int32 overflow handling
> 
> 2. Simplified context absorption logic (under review):
>    - Key insight: Most absorption opportunities provide no practical benefit
>      because subsequent contexts are discarded immediately after a match
>      succeeds. Only absorption at unbounded pattern start shows measurable
> impact.
>    - Changes:
>      * Planner (rpr.c): Added isUnboundedStart() to identify patterns where
>        leading unbounded quantifiers enable safe absorption
>      * Executor (nodeWindowAgg.c): Simplified to only absorb at pattern
> start,
>        separated cleanup logic into nfa_cleanup_dead_contexts()
>      * Added pattern->isAbsorbable flag to enable absorption only when safe
>    - Impact: Reduces context growth from quadratic to linear.
>      Example from rpr_explain.sql Test 3.1 (PATTERN A+ B, 50 rows):
>      * With absorption: 51 total contexts, 3 peak, 30 absorbed
>      * Without absorption: Would create O(n²) contexts (each row starts a
>        context, creating 1+2+3+4 contexts per 5-row segment)
>      * Absorption maintains linear O(n) growth instead of quadratic
> explosion
>    - Note: Changes in EXPLAIN ANALYZE output suggest there may be issues in
>      the matching process, particularly in absorption and NFA state
> transitions.
>      I need to review these areas more thoroughly before I can be confident
> in
>      their correctness.
> 
> 3. Added NFA statistics to EXPLAIN ANALYZE output (under review):
>    - Extended explain.c to show NFA construction and matching statistics
>    - Added comprehensive statistics tracking (execnodes.h):
>      * NFALengthStats structure for min/max/avg length measurements
>      * State statistics: peak, total created, merged (deduplicated)
>      * Context statistics: peak, total created, absorbed, skipped, pruned
>      * Match statistics: succeeded/failed counts with length distributions
>    - Implemented statistics collection in nodeWindowAgg.c
>    - New test suite: rpr_explain.sql for testing NFA statistics output
>    - Note: rpr_explain is not included in parallel_schedule (runs
> separately)
>    - Statistics reveal pattern complexity impact:
>      Example from Test 2.7 (PATTERN ((A | B)+)+, 1000 rows):
>      * NFA States: 2,664 peak, 892,447 total created
>      * Peak grows linearly with rows (~2.6x), likely due to state merging
>      * High total shows significant state churn despite controlled peak
>      * This demonstrates complex pattern behavior that users should
> understand
>      * Question: Should we add a GUC parameter to limit NFA state peak count
>        (e.g., max_rpr_nfa_states) to provide administrators with safety
> controls?
>    - This feature is still under review and may be refined in future patches
> 
> 4. Major refactoring of pattern compilation and optimization (rpr.c):
>    - Split large monolithic functions into smaller, semantically meaningful
>      single-purpose functions for better readability
>    - Pattern compilation refactored:
>      * Added collectDefineVariables() to populate variable names from DEFINE
>      * Split scanRPRPattern() into recursive worker and wrapper
>      * Extracted allocateRPRPattern() for memory allocation
>      * Split fillRPRPattern() by node type: fillRPRPatternVar(),
>        fillRPRPatternGroup(), fillRPRPatternAlt()
>    - Pattern optimization converted to dispatcher pattern:
>      * optimizeSeqPattern() - flattens nested SEQ, merges consecutive
> vars/groups
>      * optimizeAltPattern() - flattens nested ALT, removes duplicates
>      * optimizeGroupPattern() - multiplies quantifiers, unwraps single
> children
>    - Each optimization step broken into separate helper functions:
>      * flattenSeqChildren(), mergeConsecutiveVars(),
> mergeConsecutiveGroups()
>      * flattenAltChildren(), removeDuplicateAlternatives()
>      * tryMultiplyQuantifiers(), tryUnwrapGroup()
>    - New optimization: mergeGroupPrefixSuffix()
>      * A B (A B)+ -> (A B){2,}
>      * (A B)+ A B -> (A B){2,}
>      * A B (A B)+ A B -> (A B){3,}
>    - Code structure reorganized with categorized forward declarations
>    - Added test cases in rpr.sql to verify new optimizations:
>      * Consecutive GROUP merge tests
>      * Quantifier multiplication tests (including INF blocking)
>      * Group prefix/suffix merge tests
> 
> 5. Fixed quantifier validation:
>    - Corrected bounds checking in gram.y (e.g., {,m} now requires m >= 1)
>    - Improved error messages for invalid quantifier ranges
> 
> Additional notes:
> - The NFA statistics output format in EXPLAIN ANALYZE was quickly
>   implemented for debugging purposes. I would appreciate feedback from
>   experienced PostgreSQL developers on:
>   * Whether the output format and structure are appropriate
>   * Whether these statistics would be valuable for query tuning in
> production,
>     or if they are only useful for development/debugging
>   * If production-worthy, what simplifications might make them more
> accessible
>     to users unfamiliar with NFA internals
> - Reluctant quantifier handling needs further clarification. If I cannot
>   make confident fixes before the next review cycle, I may temporarily
>   raise errors for reluctant cases until the logic is properly sorted out.
> 
> Next steps:
> - The executor implementation (nodeWindowAgg.c) is the only major component
>   remaining for detailed review.
> - I will focus on thoroughly reviewing the absorption logic and NFA state
>   transition mechanisms to address the concerns noted in section 2.
> - After making necessary improvements, I will send the next patch with
>   more confidence in the executor correctness.

Ok, I have merged your patches with my patches to create v40
patches.

My patches are to fix issue with RPR + IGNORE NULLS case. I modified
ignorenulls_getfuncarginframe() which is called from window functions
when IGNORE NULLS option is specified. To cope with RPR, I added
row_is_in_reduced_frame() call to it. Also test cases were added.

Attached are the v40 patches.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  170. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-18T15:32:40Z

    Subject: [RPR] Context Absorption Design Documentation - Two-Flag Approach

Hi Hackers,

I'm writing to document the detailed design of context absorption
optimization in Row Pattern Recognition (RPR). This is mainly for
future reference - honestly, I'm afraid I'll forget these details
myself in a few months, and I'd rather have them written down
somewhere public than buried in my notes. I don't want to repeat the
mistake of famous mathematical theorems
where brilliant ideas were lost because "the margin was too narrow
to contain the proof" (yes, I'm looking at you, Fermat!). I certainly
don't want future PostgreSQL hackers to spend the next 358 years
trying to re-implement context absorption because the design rationale
was lost. Fortunately, the PostgreSQL community has provided us with
sufficient digital margins. :)

========================================================================
    ROW PATTERN RECOGNITION: CONTEXT ABSORPTION DESIGN (DRAFT)
                Two-Flag Approach for O(n²) → O(n) Optimization
========================================================================

The context absorption feature achieves O(n²) → O(n) performance
improvement by eliminating redundant match searches. The core idea
is simple: a newer context (started at a later row) can never
produce a longer match than an older context when certain conditions
are met. However, the implementation details are subtle enough that
I think they deserve careful documentation.


========================================================================
TWO-FLAG DESIGN
========================================================================

The optimization uses two element flags:

1. RPR_ELEM_ABSORBABLE - "Absorption judgment point"
   Marks WHERE to compare contexts for absorption.
   - For simple unbounded VAR (e.g., A in "A+"): the VAR itself
   - For unbounded GROUP (e.g., "(A B)+"): the END element only

2. RPR_ELEM_ABSORBABLE_BRANCH - "Absorbable region marker"
   Marks ALL elements in the absorbable region.
   - Used for efficient region tracking at runtime
   - All elements at the same depth as the unbounded start


========================================================================
WHY TWO FLAGS?
========================================================================

The key insight: for pattern "(A B)+", we cannot compare contexts
when they're at different positions (one at A, another at B). They
must synchronize at the END element.

Consider "(A B)+" matching data: A B A B A B...
  Row 0 (A): Ctx1 starts, matches A
  Row 1 (B): Ctx1 matches B → END (count=1)
  Row 2 (A): Ctx1 loops to A (count=1), Ctx2 starts, matches A (count=0)
  Row 3 (B): Ctx1 matches B → END (count=2)
             Ctx2 matches B → END (count=1)
             ↑ Both at END with same elemIdx → comparable!
             → Ctx1.count(2) >= Ctx2.count(1) → Absorb Ctx2!

Every group-length rows (2 in this case), the contexts synchronize
at END, making absorption possible. This is why we need:
- ABSORBABLE flag: to mark END as the judgment point
- ABSORBABLE_BRANCH flag: to keep isAbsorbable=true while
  traversing A→B→END


========================================================================
PATTERN-SPECIFIC FLAG SETTINGS
========================================================================

Pattern: A+ B
-------------
Element 0: A (max=INF)
  flags: ABSORBABLE | ABSORBABLE_BRANCH
Element 1: B
  flags: (none)

A+ is absorbable. Judgment point is A itself. Every row at A,
contexts can be compared and absorbed. When contexts move to B,
B has no ABSORBABLE_BRANCH flag, so isAbsorbable turns false.


Pattern: (A B)+ C
-----------------
Element 0: A (depth=1)
  flags: ABSORBABLE_BRANCH
Element 1: B (depth=1)
  flags: ABSORBABLE_BRANCH
Element 2: END (depth=1, max=INF)
  flags: ABSORBABLE | ABSORBABLE_BRANCH
Element 3: C (depth=0)
  flags: (none)

A, B, END are in the absorbable region at depth 1. Only END is the
judgment point where contexts synchronize every 2 rows (group length).
When contexts move to C (depth 0), they leave the absorbable region
and isAbsorbable turns false.


Pattern: (A+ B+)+ C
-------------------
Element 0: A (depth=2, max=INF)
  flags: ABSORBABLE | ABSORBABLE_BRANCH
Element 1: B (depth=2, max=INF)
  flags: (none)
Element 2: END (inner, depth=2)
  flags: (none)
Element 3: END (outer, depth=1, max=INF)
  flags: (none)
Element 4: C (depth=0)
  flags: (none)

Only the FIRST A+ on the FIRST iteration gets flags. Why?
  Ctx1: A+ 10 times → B+ 10 times → outer loop → C
  Ctx2: (1 row later) A+ 9 times → absorbed by Ctx1 at A
  Ctx3: (1 row later) A+ 8 times → absorbed by Ctx1 at A
  ...
  Ctx10: (1 row later) A+ 1 time → absorbed by Ctx1 at A
  Ctx11: (1 row later) B input → pattern starts at A+, no match

Judgment point is A. Absorption works there. When contexts move to B+,
it has no ABSORBABLE_BRANCH flag, so isAbsorbable turns false.


Pattern: A B C
--------------
Element 0: A
  flags: (none)
Element 1: B
  flags: (none)
Element 2: C
  flags: (none)

No unbounded element. Pattern is not absorbable.
Contexts start with isAbsorbable = false from the beginning.


Pattern: A+ C | B+
------------------
Element 0: ALT
  flags: ABSORBABLE_BRANCH
Branch 1:
  Element: A (max=INF)
    flags: ABSORBABLE | ABSORBABLE_BRANCH
  Element: C
    flags: (none)
Branch 2:
  Element: B (max=INF)
    flags: ABSORBABLE | ABSORBABLE_BRANCH

A+ branch: judgment point is A. When contexts move to C, C has no
flags, so isAbsorbable turns false.
B+ branch: judgment point is B, fully absorbable.
ALT gets ABSORBABLE_BRANCH for efficient branch-level flag management.


Pattern: A+ B | (C D)+
----------------------
Element 0: ALT
  flags: ABSORBABLE_BRANCH
Branch 1:
  Element: A (max=INF)
    flags: ABSORBABLE | ABSORBABLE_BRANCH
  Element: B
    flags: (none)
Branch 2:
  Element: C (depth=1)
    flags: ABSORBABLE_BRANCH
  Element: D (depth=1)
    flags: ABSORBABLE_BRANCH
  Element: END (depth=1, max=INF)
    flags: ABSORBABLE | ABSORBABLE_BRANCH

A+ branch: judgment point is A. When contexts move to B, B has no
flags, so isAbsorbable turns false.
(C D)+ branch: judgment point is END, fully absorbable.
Contexts on different branches cannot be compared (different elemIdx).
ALT gets ABSORBABLE_BRANCH for efficient branch-level flag management.


Pattern: A+ | B C
-----------------
Element 0: ALT
  flags: ABSORBABLE_BRANCH
Branch 1:
  Element: A (max=INF)
    flags: ABSORBABLE | ABSORBABLE_BRANCH
Branch 2:
  Element: B
    flags: (none)
  Element: C
    flags: (none)

A+ branch: judgment point is A, fully absorbable.
B C branch: no unbounded element, not absorbable. When contexts enter
B C branch, B has no ABSORBABLE_BRANCH flag, so isAbsorbable turns false.
ALT gets ABSORBABLE_BRANCH because at least one branch (A+) is absorbable.


Pattern: A+ | B C+
------------------
Element 0: ALT
  flags: ABSORBABLE_BRANCH
Branch 1:
  Element: A (max=INF)
    flags: ABSORBABLE | ABSORBABLE_BRANCH
Branch 2:
  Element: B
    flags: (none)
  Element: C (max=INF)
    flags: (none)

A+ branch: judgment point is A, fully absorbable.
B C+ branch: B is not absorbable, C+ has no flags (not first unbounded).


Pattern: A B | C D
------------------
Element 0: ALT
  flags: (none)
Branch 1:
  Element: A
    flags: (none)
  Element: B
    flags: (none)
Branch 2:
  Element: C
    flags: (none)
  Element: D
    flags: (none)

Neither branch has unbounded elements. Pattern is not absorbable.
Contexts start with isAbsorbable = false from the beginning.
ALT has no flags because no branch is absorbable.


========================================================================
ABSORBABLE REGION BOUNDARY: When Contexts Leave Absorbable Region
========================================================================

This section illustrates when contexts transition from absorbable to
non-absorbable state by showing how hasAbsorbableState flag changes
as contexts progress through pattern elements.

Pattern: A+ B+
--------------
Element 0: A (max=INF)
  flags: ABSORBABLE | ABSORBABLE_BRANCH
Element 1: B (max=INF)
  flags: (none)

Context execution:
  Row 0: Ctx starts at A
    → state at A (count=1) → state.isAbsorbable = true
    → Ctx.hasAbsorbableState = true (at least one state is absorbable)
    → Ctx.allStatesAbsorbable = true (all states are absorbable)

  Row 1: Ctx at A
    → state at A (count=2) → state.isAbsorbable = true
    → Ctx.hasAbsorbableState = true
    → Ctx.allStatesAbsorbable = true

  Row 2: First B input, Ctx progresses from A to B
    → state transitions to B (count=1) → state.isAbsorbable = false
    → B has no ABSORBABLE_BRANCH flag
    → Ctx.hasAbsorbableState = false (no absorbable states remain)
    → Ctx.allStatesAbsorbable = false
    → Monotonic property: hasAbsorbableState stays false forever

  Row 3+: Ctx continues at B
    → Ctx.hasAbsorbableState = false (permanent)
    → This context can never absorb others, and cannot be absorbed

Key point: Once contexts leave A+ region and enter B+, absorption
completely stops. The A+ portion achieved O(n) absorption while it
lasted, which is the design goal.


Pattern: ((A B)+ C)+
--------------------
Outer loop iteration 1:
  Inner (A B)+: Elements A, B, inner END (depth=1)
    flags on A, B: ABSORBABLE_BRANCH
    flags on inner END: ABSORBABLE | ABSORBABLE_BRANCH
  Element C (depth=0):
    flags: (none)
  Outer END (depth=0):
    flags: (none)

Context execution (first outer iteration):
  First inner iteration of (A B)+:
    Row 0: Ctx starts at A
      → state at A (count=0, depth=1) → state.isAbsorbable = true
      → Ctx.hasAbsorbableState = true (at least one state is absorbable)
      → Ctx.allStatesAbsorbable = true (all states are absorbable)

    Row 1: Ctx at B, then epsilon transition to inner END
      → state at B (count=0, depth=1) → state.isAbsorbable = true
      → B→END is epsilon transition (same row processing)
      → state at inner END (count=1, depth=1) → state.isAbsorbable = true
      → Ctx.hasAbsorbableState = true
      → Ctx.allStatesAbsorbable = true
      → Inner END loops back to A

  Second inner iteration of (A B)+:
    Row 2: Ctx at A again
      → state at A (count=1, depth=1) → state.isAbsorbable = true
      → Ctx.hasAbsorbableState = true
      → Ctx.allStatesAbsorbable = true

    Row 3: Ctx at B, then epsilon transition to inner END → exit to C
      → state at B (count=1, depth=1) → state.isAbsorbable = true
      → state at inner END (count=2, depth=1) → state.isAbsorbable = true
      → Inner END can loop back or exit (assume (A B)+ min satisfied, exit
to C)
      → Ctx.hasAbsorbableState = true (still in absorbable region)
      → Ctx.allStatesAbsorbable = true

  Row 4: Ctx at C, then epsilon transition to outer END
    → state transitions to C (depth=0) → state.isAbsorbable = false
    → C has no ABSORBABLE_BRANCH flag (depth=0, outside inner group)
    → Ctx.hasAbsorbableState = false (no absorbable states remain)
    → Ctx.allStatesAbsorbable = false (C state is not absorbable)
    → Monotonic property: hasAbsorbableState = false is PERMANENT for this
context
    → C→outer END is epsilon transition (same row processing)
    → state at outer END (count=1, depth=0) → state.isAbsorbable = false
    → Ctx.hasAbsorbableState = false (still false)
    → Ctx.allStatesAbsorbable = false (still false)
    → Outer END loops back to start

  Row 5+: Ctx continues in second outer iteration (same context)
    → Outer END loops back to A (depth=1)
    → New A state created via transition from outer END state
    → state.isAbsorbable = false (inherited from outer END state,
monotonic!)
    → A element has ABSORBABLE_BRANCH flag, but cannot override transition
inheritance
    → Ctx.hasAbsorbableState = false (PERMANENT, monotonic property!)
    → Ctx.allStatesAbsorbable = false (PERMANENT, A state is not absorbable)
    → This context can no longer absorb others (hasAbsorbableState = false)
    → This context cannot be absorbed (allStatesAbsorbable = false)
    → Context continues matching but absorption is disabled forever

Note: Element flags only apply at initial state creation (context start).
During state transitions, state.isAbsorbable is inherited from previous
state.
Once false, it remains false through all subsequent transitions.

Key points:
- Within first outer iteration's (A B)+ region, absorption works
- When progressing to C (Row 4), hasAbsorbableState becomes false
permanently
- Same context continuing to second iteration: absorption stays disabled
- NEW contexts starting at later rows: fresh start with hasAbsorbableState
= true
- Example: If a new context (Ctx2) starts at Row 5, Ctx2.hasAbsorbableState
= true
- Pattern achieves O(n) absorption within first outer iteration's (A B)+
portion


========================================================================
DETAILED EXECUTION EXAMPLE: A+ | B C+
========================================================================

This section provides a detailed walkthrough of the "A+ | B C+" pattern
execution to illustrate absorption eligibility restoration, state
transition design, and the two-flag absorption mechanism.

Pattern Structure:
------------------
Element 0: ALT
  flags: ABSORBABLE_BRANCH
Branch 1:
  Element: A (max=INF)
    flags: ABSORBABLE | ABSORBABLE_BRANCH
Branch 2:
  Element: B
    flags: (none)
  Element: C (max=INF)
    flags: (none)

Execution with input A=TRUE, B=TRUE, C=FALSE repeated:

Note: A+ keeps A state at the last matched position (not FIN state).
FIN transition is checked and recorded in matchedState, but FIN state
itself is not added to ctx->states. This keeps ctx->states clean with
only in-progress states for absorption comparison.

  Row 1: Ctx1 starts → ALT creates A state and B state simultaneously
    → A state: A=TRUE matches (count=1) [state.isAbsorbable=true]
         * Can transition to FIN → Ctx1.matchedState recorded (Row 1)
         * A state remains in ctx->states (for potential continuation)
    → B state: B=TRUE matches (count=1) [state.isAbsorbable=false]
         * B is bounded (max=1), but remains at B state for next row check
         * B state remains in ctx->states (will check repetition on next
row)
    → Row 1 end: Ctx1.states = {A state (isAbsorbable=true), B state
(isAbsorbable=false)}
    → Ctx1.hasAbsorbableState = true (A state is absorbable)
    → Ctx1.allStatesAbsorbable = false (B state is not absorbable)
    → Ctx1.matchedState = A branch match ending at Row 1

  Row 2: Ctx2 starts (same as Ctx1 at Row 1)
    → Ctx1: A state matches A=TRUE (count=2) [isAbsorbable=true]
         * Can transition to FIN → Ctx1.matchedState updated (Row 2, longer
match)
         * A state remains in ctx->states (count=2)
         B state: count=1 already, max=1, cannot repeat → transitions to C
state
         C state checks C=FALSE → mismatch, dies
         * Note: Even if C+ completed, FIN check unnecessary
           (A+ matchedState has better altPriority, would win anyway)
    → Ctx1 Row 2 end: Ctx1.states = {A state (isAbsorbable=true, count=2)}
    → Ctx1.hasAbsorbableState = true (A state is absorbable)
    → Ctx1.allStatesAbsorbable = true (only A state, which is absorbable)
    → Ctx2: A state (count=1) [isAbsorbable=true]
         * Can transition to FIN → Ctx2.matchedState recorded (Row 2)
         * A state remains in ctx->states
         B state: B=TRUE matches (count=1) [isAbsorbable=false]
         * B state remains (will transition to C on next row)
    → Ctx2 Row 2 end: Ctx2.states = {A state (isAbsorbable=true), B state
(isAbsorbable=false)}
    → Ctx2.hasAbsorbableState = true (A state is absorbable)
    → Ctx2.allStatesAbsorbable = false (B state is not absorbable)
    → Absorption check: Ctx1.hasAbsorbableState && Ctx2.allStatesAbsorbable?
    → false (Ctx2.allStatesAbsorbable = false) → No absorption

  Row 3: Ctx3 starts
    → Ctx1: A state matches A=TRUE (count=3) [isAbsorbable=true]
         * Can transition to FIN → Ctx1.matchedState updated (Row 3)
         * A state remains (count=3)
    → Ctx1 Row 3 end: Ctx1.states = {A state (isAbsorbable=true, count=3)}
    → Ctx1.hasAbsorbableState = true
    → Ctx1.allStatesAbsorbable = true
    → Ctx2: A state matches A=TRUE (count=2) [isAbsorbable=true]
         * Can transition to FIN → Ctx2.matchedState updated (Row 3)
         * A state remains (count=2)
         B state: count=1 already, max=1, cannot repeat → transitions to C
state
         C state checks C=FALSE → mismatch, dies
         * Note: Even if C+ completed, FIN check unnecessary (lexical order)
    → Ctx2 Row 3 end: Ctx2.states = {A state (isAbsorbable=true, count=2)}
    → Ctx2.hasAbsorbableState = true (A state is absorbable)
    → Ctx2.allStatesAbsorbable = true (only A state, which is absorbable)
    → Ctx3: A state (count=1) [isAbsorbable=true]
         B state: B=TRUE matches (count=1) [isAbsorbable=false]
    → Ctx3 Row 3 end: Ctx3.states = {A state (isAbsorbable=true), B state
(isAbsorbable=false)}
    → Ctx3.hasAbsorbableState = true (A state is absorbable)
    → Ctx3.allStatesAbsorbable = false (B state is not absorbable)
    → Absorption check: Ctx1 vs Ctx3 (checked first, from back)
         * Pre-check: Ctx1.hasAbsorbableState && Ctx3.allStatesAbsorbable?
         * false (Ctx3.allStatesAbsorbable = false)
         * No absorption attempted (fast filter rejected)
    → Absorption check: Ctx1 vs Ctx2
         * Pre-check: Ctx1.hasAbsorbableState && Ctx2.allStatesAbsorbable?
         * true (both flags are true)
         * Ctx1.states = {A state (count=3)}
         * Ctx2.states = {A state (count=2)}
         * All Ctx2 states covered by Ctx1 (A: 3 >= 2) → Ctx2 absorbed!
         * Note: Even if Ctx1 had additional states (absorbable or not),
           Ctx2 would still be absorbed as long as all Ctx2 states are
covered

This example demonstrates three key design aspects:

1. Two-flag absorption design: Ctx2 becomes eligible for absorption
   when its B state disappears due to mismatch. Two distinct flags track:
   - hasAbsorbableState: remains true throughout (A state always absorbable)
   - allStatesAbsorbable: changes from false (Row 2, has B) to true (Row 3,
only A)
   This separation enables fast pre-check without state iteration.

2. State transition design: A+ keeps A state in ctx->states at the
   last matched position for absorption comparison. FIN transition is
   only checked and recorded in matchedState, not added to ctx->states.

3. Asymmetric absorption logic: Ctx1 can absorb Ctx2 even if Ctx1
   has additional states, as long as Ctx2.allStatesAbsorbable=true and
   all Ctx2 states are covered by matching Ctx1 states with higher counts.

4. Pruning opportunity (future optimization): With eager pruning,
   the B state could be eliminated at Row 1 (since max=1, count=1, no
   further progression possible), making Ctx1.allStatesAbsorbable=true
   immediately. This would enable absorption at Row 2 instead of Row 3.
   Alternatively, when B state dies during Row 2 processing, we could
   immediately update allStatesAbsorbable and trigger absorption without
   waiting for the next row. However, this optimization is not
   implemented yet - implementation is left for future work.


========================================================================
STATE TRANSITION DESIGN FOR ABSORPTION
========================================================================

NFA State Management Philosophy
--------------------------------

The absorption optimization requires clean separation between:
- In-progress states (ctx->states): active states being evaluated each row
- Completion markers (ctx->matchedState): records best match found so far

This design decision has profound implications for how state transitions
work, especially for unbounded quantifiers like A+.


The FIN State Problem
---------------------

Traditional NFA implementation for A+ would:
  1. Match A=TRUE → increment count
  2. Create TWO states:
     - Loop state: A (count++) → add to ctx->states for next row
     - Escape state: FIN (pattern complete) → add to ctx->states

Problem: If FIN state is added to ctx->states, absorption becomes complex:
  - Ctx1.states = {A state (count=3), FIN state}
  - Ctx2.states = {A state (count=2), FIN state}
  - Both have FIN states, but should we compare them?
  - FIN states don't have meaningful counts to compare
  - FIN is not in ABSORBABLE region (pattern already finished)


The Solution: Deferred FIN Transition
--------------------------------------

Instead of creating FIN state in ctx->states, we:
  1. Keep A state at last matched position in ctx->states
  2. Check if FIN transition is possible
  3. Record completion in ctx->matchedState (separate from ctx->states)
  4. FIN state is NOT added to ctx->states

Example for A+ with continuous A=TRUE input:
  Row 1: Ctx1.states = {A state (count=1)}
         Ctx1.matchedState = {FIN, ending at Row 1}
  Row 2: Ctx1.states = {A state (count=2)}  ← A state updated, not
duplicated
         Ctx1.matchedState = {FIN, ending at Row 2}  ← updated to longer
match
  Row 3: Ctx1.states = {A state (count=3)}
         Ctx1.matchedState = {FIN, ending at Row 3}

Absorption comparison (Row 3):
  - Ctx1.states = {A state (count=3)}
  - Ctx2.states = {A state (count=2)}
  - Compare: count=3 >= count=2, same elemIdx → absorb Ctx2
  - Clean! No FIN states to complicate the comparison.


Benefits of This Design
------------------------

1. Clean absorption logic:
   - ctx->states contains ONLY in-progress states
   - All states in ctx->states are comparable (same depth, count meaningful)
   - FIN states don't interfere with absorption judgment

2. Efficient state management:
   - No duplicate FIN states created each row
   - matchedState updated in-place (not accumulated)
   - Reduced memory footprint

3. Correct greedy semantics:
   - A+ continues matching as long as A=TRUE
   - matchedState always records the longest match so far
   - When A finally mismatches, we use the last matchedState

4. Simple state tracking:
   - ctx->states reflects "where we are in the pattern"
   - matchedState reflects "best completion found so far"
   - No confusion between "still matching" vs "already finished"


State Absorbability Inheritance
--------------------------------

Element flags (ABSORBABLE, ABSORBABLE_BRANCH) only apply at initial state
creation when a context starts. During state transitions, state.isAbsorbable
is inherited from the previous state, not from element flags.

This inheritance mechanism has critical implications:

1. Initial state creation (context start at Row 0):
   - state.isAbsorbable = (elem->flags & RPR_ELEM_ABSORBABLE_BRANCH) != 0
   - Element flags determine initial absorbability

2. State transitions (A→B, B→END, END→A, etc.):
   - New state inherits state.isAbsorbable from previous state
   - Element flags on the new element do NOT override this inheritance
   - Once state.isAbsorbable = false, it remains false through all
transitions

3. Monotonic property:
   - state.isAbsorbable can change from true → false (leaving absorbable
region)
   - state.isAbsorbable CANNOT change from false → true (cannot re-enter)
   - This ensures contexts permanently leave absorbable region once they
exit

Example: Pattern ((A B)+ C)+

First outer iteration (Row 0-4):
  Row 0: Context starts → A state created
    → state.isAbsorbable = true (from A element's ABSORBABLE_BRANCH flag)

  Row 1: A→B transition
    → B state.isAbsorbable = true (inherited from A state, not from B flag)

  Row 1: B→END transition (epsilon, same row)
    → END state.isAbsorbable = true (inherited from B state)

  Row 4: C created via transition from inner END
    → C state.isAbsorbable = false (C element has no ABSORBABLE_BRANCH flag)

  Row 4: C→outer END transition (epsilon, same row)
    → outer END state.isAbsorbable = false (inherited from C state)

Second outer iteration (Row 5+, same context continuing):
  Row 5+: outer END loops back to A
    → New A state created via transition from outer END state
    → state.isAbsorbable = false (INHERITED from outer END state)
    → A element has ABSORBABLE_BRANCH flag, but this does NOT apply
    → Element flags only apply at context start, not during transitions
    → Monotonic property: once false, stays false forever

This is why hasAbsorbableState becomes permanently false when contexts
leave the first unbounded region - all subsequent states inherit
isAbsorbable=false regardless of element flags they transition to.


========================================================================
PLANNING STAGE: computeAbsorbability()
========================================================================

Located in src/backend/optimizer/plan/rpr.c.

Core Concept: Find First Unbounded Portion
-------------------------------------------
The algorithm identifies the FIRST unbounded portion of the pattern starting
from element 0. Only this first portion gets absorption flags. Elements
after
this portion do not get flags, even if they are unbounded.

Examples:
- A+ B+: First unbounded portion is A+. B+ gets no flags.
- ((A B)+ C)+: First unbounded portion is (A B)+. C and outer END get no
flags.
- (A B)+ C: First unbounded portion is (A B)+. C gets no flags.

Key constraints for the first unbounded portion:
- Must be unbounded VAR (max=INF) or unbounded GROUP (END has max=INF)
- For unbounded GROUP:
  * All elements inside at the same depth must be simple {1,1} VARs
  * No nested GROUPs inside (depth > startDepth)
  * No unbounded VARs inside (all VARs must be {1,1})

Why these constraints:
- Simple VARs ensure predictable synchronization at END element
- No nested GROUPs keeps depth-based flag assignment simple
- No unbounded VARs inside prevents complex state combinations

Examples of valid first unbounded portions:
- A+ (simple unbounded VAR)
- (A B)+ (unbounded GROUP with simple VARs A and B at depth=1)
- (A B C)+ (unbounded GROUP with simple VARs A, B, C at depth=1)

Examples where outer pattern structure is complex, but absorption still
works:
- ((A B)+ C)+
  * Analysis STOPS at first unbounded portion: (A B)+
  * First portion (A B)+ is valid: A and B are simple {1,1} VARs at depth=1
  * C and outer END are beyond first portion (ignored, get no flags)
  * Result: Absorption works for the (A B)+ portion

Examples that fail constraints (cannot set absorption flags):
- (A+ B+)+ (inner A+ and B+ are unbounded VARs, violates constraint for
first portion)

Algorithm:

1. Check if pattern is too short (< 2 elements) → not absorbable

2. If pattern starts with ALT:
   - Iterate through each branch
   - For each branch, check isUnboundedStart() on first element of branch
   - If true, call setAbsorbabilityFlagsForBranch() for that branch
   - If any branch is absorbable, set ABSORBABLE_BRANCH on ALT element

3. If pattern does not start with ALT:
   - Check isUnboundedStart() on first element (element 0)
   - If true, call setAbsorbabilityFlagsForBranch(pattern, 0)

Helper function isUnboundedStart(pattern, idx):
  Scans from idx forward to determine if this starts an unbounded portion.

  Case 1: Simple unbounded VAR
    - Element at idx has max=INF
    - Return true immediately

  Case 2: Unbounded GROUP
    - Scan elements from idx forward while depth >= startDepth
    - Find END element with:
      * depth == startDepth
      * max == INF (unbounded loop)
      * jump == idx (loops back to start)
    - Check all elements at startDepth are simple {1,1} VARs
    - Check no nested structures (depth > startDepth)
    - Check for outer nesting: if another END exists at lower depth before
FIN
      → nested structure, return false
    - Return true if all checks pass

Helper function setAbsorbabilityFlagsForBranch(pattern, branchIdx):
  Performs two passes:

  1. First pass: detect if this is an unbounded GROUP pattern
     - Scan elements from branchIdx to find END with unbounded max that
jumps back
     - Record endIdx if found

  2. Second pass: set flags on all elements at unbounded depth
     - ABSORBABLE_BRANCH: all elements where depth == startDepth
     - ABSORBABLE: judgment point only
       * Unbounded GROUP → only END element gets ABSORBABLE
       * Simple unbounded VAR → only first element gets ABSORBABLE

Runtime conditions (in buildRPRPattern):
  - Check runtime conditions first to avoid unnecessary pattern analysis:
    * rpSkipTo must be ST_PAST_LAST_ROW (not NEXT ROW)
    * Frame must be unlimited (UNBOUNDED FOLLOWING or not ROWS mode)
  - If conditions met: call computeAbsorbability() for structural check
  - If conditions not met: set pattern->isAbsorbable = false directly
  - This eliminates runtime condition checks in executor


========================================================================
RUNTIME OPERATION
========================================================================

Located in src/backend/executor/nodeWindowAgg.c.

State-Level and Context-Level Absorption Flags:

Each NFA state tracks its own absorption status (state->isAbsorbable).
Context tracks TWO distinct absorption flags for different purposes:

State Initialization:
  Two different mechanisms depending on how state is created:

  1. Initial state creation (in nfa_start_context):
     - state->isAbsorbable = (elem->flags & RPR_ELEM_ABSORBABLE_BRANCH) != 0
     - Element flags determine initial absorbability when context starts

  2. State transitions (in nfa_state_clone):
     - state->isAbsorbable = previousState->isAbsorbable
     - Inherited from previous state, NOT from element flags
     - Element flags are NOT checked during transitions
     - Monotonic: once false, stays false through all transitions

Context Initialization (in nfa_start_context):
  - ctx->hasAbsorbableState = pattern->isAbsorbable
  - ctx->allStatesAbsorbable = pattern->isAbsorbable
  - Initial state's isAbsorbable also set based on element 0's flag

Flag Management (after each row in nfa_step()):
  Two flags track absorption status:

  1. hasAbsorbableState (can this context absorb others?):
     - true if at least one state has isAbsorbable=true
     - false if all states have isAbsorbable=false
     - Indicates context is in absorbable region
     - Monotonic: once false, stays false forever

  2. allStatesAbsorbable (can this context be absorbed?):
     - true if ALL states have isAbsorbable=true
     - false if any state has isAbsorbable=false
     - Indicates context is eligible for absorption
     - Can change: false → true (non-absorbable states die)

  Optimization: Once hasAbsorbableState becomes false, no need to
recalculate:
    - All absorbable states have died
    - New states only come from transitions (can't create new absorbable
states)
    - Both flags remain false permanently
    - Skip all state iteration for flag updates

  Recalculation logic (only if hasAbsorbableState still true):
    - Iterate through all states
    - Check each state's isAbsorbable flag
    - Update both flags based on current states

Absorption Check (in nfa_absorb_contexts()):
  Runtime conditions (SKIP TO PAST LAST ROW, unlimited frame) are
  pre-validated at planning time. Executor simply checks
pattern->isAbsorbable.

  Absorption logic: Can Ctx1 (older) absorb Ctx2 (newer)?

  Pre-check (fast filter using context flags):
  1. Ctx1.hasAbsorbableState == true (Ctx1 has ability to absorb)
  2. Ctx2.allStatesAbsorbable == true (Ctx2 is eligible to be absorbed)
     - If either check fails → skip absorption entirely
     - No need to iterate through states for these checks

  Actual absorption check (if pre-check passes):
  3. For each state in Ctx2, find matching state in Ctx1 (same elemIdx)
  4. Each Ctx2 state must be covered: Ctx1 count >= Ctx2 count

  Key insight: asymmetric check with two-flag design
  - Ctx2 (newer): allStatesAbsorbable must be true (checked via flag)
  - Ctx1 (older): hasAbsorbableState must be true (may have extra
non-absorbable states)
  - We check if all Ctx2 states are covered by matching Ctx1 states

  Example:
  - Ctx1.states = {A state (absorbable, count=3), X state (non-absorbable)}
  - Ctx1.hasAbsorbableState = true, Ctx1.allStatesAbsorbable = false
  - Ctx2.states = {A state (absorbable, count=2)}
  - Ctx2.hasAbsorbableState = true, Ctx2.allStatesAbsorbable = true
  - Pre-check: Ctx1.hasAbsorbableState && Ctx2.allStatesAbsorbable? Yes
  - Check: A state in Ctx1 covers A state in Ctx2? Yes (3 >= 2)
  - Result: Ctx2 absorbed
  - Ctx1's X state doesn't prevent absorption

This design ensures:
  - No repeated element flag lookups (cached in state)
  - No runtime condition checks (validated once at planning)
  - Fast context-level filtering (two pre-computed flags, no state
iteration)
  - Correct synchronization (only at designated judgment points)
  - Simple depth-independent logic (flag-based, not depth comparisons)
  - Asymmetric check: newer context must be fully absorbable, older just
needs one


========================================================================
PERFORMANCE IMPACT
========================================================================

Test results for "(A B)+" pattern with 60 rows:
- Absorbed length: 2/2/2 (= group length, correct synchronization)
- Peak states: 5
- Peak contexts: 4
- Absorbed count: 29

The two-flag design ensures contexts synchronize at END elements,
preventing premature absorption attempts at intermediate positions
(A or B). This guarantees correctness while maintaining efficiency.

For absorbable patterns like "A+" or "(A B)+", practically ALL
eligible contexts get absorbed because:
- Contexts separated by group-length rows always synchronize at
  judgment points
- Older contexts always have count >= newer contexts at the same
  judgment point
- This is exactly what achieves O(n²) → O(n) reduction


========================================================================
SUMMARY
========================================================================

The context absorption optimization achieves O(n²) → O(n) performance
by eliminating redundant match searches. The design uses multiple
coordinated mechanisms:

1. Two-Flag Element Design (Planning Time):
   - RPR_ELEM_ABSORBABLE: marks judgment points (WHERE to compare)
   - RPR_ELEM_ABSORBABLE_BRANCH: marks absorbable regions (tracks
boundaries)
   - For simple unbounded VAR (A+): judgment point is the VAR itself
   - For unbounded GROUP ((A B)+): judgment point is END element only
   - Ensures contexts synchronize before comparison

2. First Unbounded Portion Strategy (Planning Time):
   - Algorithm identifies FIRST unbounded portion starting from element 0
   - Only this portion gets absorption flags
   - Examples: A+ B+ → only A+ flagged; ((A B)+ C)+ → only (A B)+ flagged
   - Achieves O(n) absorption within first portion (sufficient for most
cases)

3. State Inheritance Mechanism (Runtime):
   - Element flags apply ONLY at context start (nfa_start_context)
   - State transitions inherit state.isAbsorbable from previous state
   - Element flags do NOT override inheritance during transitions
   - Monotonic: once state.isAbsorbable = false, stays false forever
   - This ensures contexts permanently leave absorbable region after first
portion

4. Two-Flag Context Design (Runtime):
   - hasAbsorbableState: can this context absorb others? (≥1 absorbable
state)
     * Monotonic: true→false only, cannot recover once false
   - allStatesAbsorbable: can this context be absorbed? (ALL states
absorbable)
     * Dynamic: can change false→true (when non-absorbable states die)
   - Enables fast pre-check without state iteration
   - Optimization: once hasAbsorbableState = false, skip all recalculation
forever

5. FIN State Separation (Runtime):
   - ctx->states: in-progress states only (for absorption comparison)
   - ctx->matchedState: pattern completion marker (best match so far)
   - FIN state NOT added to ctx->states (keeps comparison clean)
   - A+ pattern: A state remains at last position, FIN only in matchedState
   - Prevents FIN states from complicating absorption logic

6. Asymmetric Absorption Logic (Runtime):
   - Pre-check: Ctx1.hasAbsorbableState && Ctx2.allStatesAbsorbable
   - Older context (Ctx1): needs ≥1 absorbable state (may have extra states)
   - Newer context (Ctx2): ALL states must be absorbable
   - Check: all Ctx2 states covered by matching Ctx1 states (count ≥)
   - Asymmetry enables absorption even when Ctx1 has non-absorbable states

7. Synchronization at Judgment Points (Runtime):
   - Contexts only compared when at same elemIdx (same pattern position)
   - For (A B)+: contexts synchronize at END every group-length rows
   - For A+: contexts synchronize at A every row
   - Prevents premature comparison at intermediate positions
   - Guarantees correctness while maintaining O(n) efficiency

Key Benefits:
- Correct synchronization: contexts compared only at designated judgment
points
- Efficient runtime: no repeated element flag lookups, fast context-level
filtering
- Simple state management: clean separation between in-progress vs
completed states
- Depth-independent logic: flag-based design works for nested patterns
- Monotonic optimization: permanent false flags skip unnecessary
recalculation
- No runtime condition checks: pre-validated at planning time


========================================================================
FUTURE WORK
========================================================================

Element Flags for Next Element Type
------------------------------------
To optimize state transition decisions, planning-time flags indicate
next element type (defined in src/include/optimizer/rpr.h):

  RPR_ELEM_NEXT_IS_END: next element is END (epsilon transition, loop back)
  RPR_ELEM_NEXT_IS_FIN: next element is FIN (epsilon transition, pattern
end)

These flags enable faster runtime decisions without accessing the next
element:
  - Faster state transition decisions (single flag check vs array lookup)
  - Better cache locality (element data already in cache)
  - Clear distinction between END and FIN transitions
  - Simplified runtime logic (flag-based branching)

Flag definitions added to header file. Setter logic in planning code (rpr.c)
would scan elements and set flags based on next element type during pattern
compilation.


Reluctant Quantifier Support (Mid-term)
---------------------------------------
Reluctant (non-greedy) quantifiers (A+?, B+?) are defined in SQL:2016
standard
but not yet supported by the PostgreSQL parser. Pattern syntax containing
'?'
after quantifiers currently results in parser error.

The RPR_ELEM_RELUCTANT flag (0x01) is defined in the header file for future
use, preserving the grammar structure for when parser support is added.

Once parser support is implemented, absorption optimization considerations:
  - Reluctant quantifiers may create different synchronization patterns
  - Count comparison logic remains valid (Ctx1 count >= Ctx2 count)
  - May benefit from specialized absorption rules

This is left as a mid-term consideration pending parser implementation and
subsequent analysis of interaction between reluctant semantics and
absorption.


Lexical Order Optimization for ALT Patterns (Long-term)
--------------------------------------------------------
In ALT patterns (e.g., "A+ | B C+"), once a higher-priority branch creates
a matchedState, lower-priority branches cannot win due to lexical order.

Optimization opportunity:
  - Once A+ branch creates matchedState, B C+ branch cannot win
  - B C+ branch can skip FIN transition checks entirely
  - Early pruning: if ctx->matchedState exists with better altPriority,
    skip completion checks for lower-priority branches

Current implementation processes all branches fully, which is correct but
does redundant work for lower-priority branches. This optimization is left
as a long-term consideration due to implementation complexity versus
marginal performance gains.


I hope this documentation helps anyone (including future me) who
needs to understand or modify this code. The design took several
iterations to get right, and I wanted to record the reasoning
while it's still fresh in my mind.

Comments welcome!

Best regards,
Henson

2026년 1월 18일 (일) PM 9:09, Tatsuo Ishii <ishii@postgresql.org>님이 작성:

> Hi Henson,
>
> >  Attached is the patch with the following changes:
> >
> > 1. Removed 26 variables limit and fixed count overflow:
> >    - Deleted defineInitial field from WindowClause (parsenodes.h),
> >      WindowAgg (plannodes.h), and related code in createplan.c and
> > parse_rpr.c
> >    - Now uses RPR_VARID_MAX (252) constant for the limit check
> >    - Fixed NFA count overflow issue:
> >      * Changed RPRNFAState.counts from int16 to int32 (execnodes.h)
> >      * Added RPR_COUNT_MAX (INT32_MAX) constant (rpr.h)
> >      * Handles pattern matching failures correctly at >32,767 repetitions
> >      * Added INT32_MAX boundary handling to prevent overflow
> >    - Updated test cases in rpr.sql:
> >      * Updated variable limit tests from 26 to 252
> >      * Added 100K row large partition test to verify int32 overflow
> handling
> >
> > 2. Simplified context absorption logic (under review):
> >    - Key insight: Most absorption opportunities provide no practical
> benefit
> >      because subsequent contexts are discarded immediately after a match
> >      succeeds. Only absorption at unbounded pattern start shows
> measurable
> > impact.
> >    - Changes:
> >      * Planner (rpr.c): Added isUnboundedStart() to identify patterns
> where
> >        leading unbounded quantifiers enable safe absorption
> >      * Executor (nodeWindowAgg.c): Simplified to only absorb at pattern
> > start,
> >        separated cleanup logic into nfa_cleanup_dead_contexts()
> >      * Added pattern->isAbsorbable flag to enable absorption only when
> safe
> >    - Impact: Reduces context growth from quadratic to linear.
> >      Example from rpr_explain.sql Test 3.1 (PATTERN A+ B, 50 rows):
> >      * With absorption: 51 total contexts, 3 peak, 30 absorbed
> >      * Without absorption: Would create O(n²) contexts (each row starts a
> >        context, creating 1+2+3+4 contexts per 5-row segment)
> >      * Absorption maintains linear O(n) growth instead of quadratic
> > explosion
> >    - Note: Changes in EXPLAIN ANALYZE output suggest there may be issues
> in
> >      the matching process, particularly in absorption and NFA state
> > transitions.
> >      I need to review these areas more thoroughly before I can be
> confident
> > in
> >      their correctness.
> >
> > 3. Added NFA statistics to EXPLAIN ANALYZE output (under review):
> >    - Extended explain.c to show NFA construction and matching statistics
> >    - Added comprehensive statistics tracking (execnodes.h):
> >      * NFALengthStats structure for min/max/avg length measurements
> >      * State statistics: peak, total created, merged (deduplicated)
> >      * Context statistics: peak, total created, absorbed, skipped, pruned
> >      * Match statistics: succeeded/failed counts with length
> distributions
> >    - Implemented statistics collection in nodeWindowAgg.c
> >    - New test suite: rpr_explain.sql for testing NFA statistics output
> >    - Note: rpr_explain is not included in parallel_schedule (runs
> > separately)
> >    - Statistics reveal pattern complexity impact:
> >      Example from Test 2.7 (PATTERN ((A | B)+)+, 1000 rows):
> >      * NFA States: 2,664 peak, 892,447 total created
> >      * Peak grows linearly with rows (~2.6x), likely due to state merging
> >      * High total shows significant state churn despite controlled peak
> >      * This demonstrates complex pattern behavior that users should
> > understand
> >      * Question: Should we add a GUC parameter to limit NFA state peak
> count
> >        (e.g., max_rpr_nfa_states) to provide administrators with safety
> > controls?
> >    - This feature is still under review and may be refined in future
> patches
> >
> > 4. Major refactoring of pattern compilation and optimization (rpr.c):
> >    - Split large monolithic functions into smaller, semantically
> meaningful
> >      single-purpose functions for better readability
> >    - Pattern compilation refactored:
> >      * Added collectDefineVariables() to populate variable names from
> DEFINE
> >      * Split scanRPRPattern() into recursive worker and wrapper
> >      * Extracted allocateRPRPattern() for memory allocation
> >      * Split fillRPRPattern() by node type: fillRPRPatternVar(),
> >        fillRPRPatternGroup(), fillRPRPatternAlt()
> >    - Pattern optimization converted to dispatcher pattern:
> >      * optimizeSeqPattern() - flattens nested SEQ, merges consecutive
> > vars/groups
> >      * optimizeAltPattern() - flattens nested ALT, removes duplicates
> >      * optimizeGroupPattern() - multiplies quantifiers, unwraps single
> > children
> >    - Each optimization step broken into separate helper functions:
> >      * flattenSeqChildren(), mergeConsecutiveVars(),
> > mergeConsecutiveGroups()
> >      * flattenAltChildren(), removeDuplicateAlternatives()
> >      * tryMultiplyQuantifiers(), tryUnwrapGroup()
> >    - New optimization: mergeGroupPrefixSuffix()
> >      * A B (A B)+ -> (A B){2,}
> >      * (A B)+ A B -> (A B){2,}
> >      * A B (A B)+ A B -> (A B){3,}
> >    - Code structure reorganized with categorized forward declarations
> >    - Added test cases in rpr.sql to verify new optimizations:
> >      * Consecutive GROUP merge tests
> >      * Quantifier multiplication tests (including INF blocking)
> >      * Group prefix/suffix merge tests
> >
> > 5. Fixed quantifier validation:
> >    - Corrected bounds checking in gram.y (e.g., {,m} now requires m >= 1)
> >    - Improved error messages for invalid quantifier ranges
> >
> > Additional notes:
> > - The NFA statistics output format in EXPLAIN ANALYZE was quickly
> >   implemented for debugging purposes. I would appreciate feedback from
> >   experienced PostgreSQL developers on:
> >   * Whether the output format and structure are appropriate
> >   * Whether these statistics would be valuable for query tuning in
> > production,
> >     or if they are only useful for development/debugging
> >   * If production-worthy, what simplifications might make them more
> > accessible
> >     to users unfamiliar with NFA internals
> > - Reluctant quantifier handling needs further clarification. If I cannot
> >   make confident fixes before the next review cycle, I may temporarily
> >   raise errors for reluctant cases until the logic is properly sorted
> out.
> >
> > Next steps:
> > - The executor implementation (nodeWindowAgg.c) is the only major
> component
> >   remaining for detailed review.
> > - I will focus on thoroughly reviewing the absorption logic and NFA state
> >   transition mechanisms to address the concerns noted in section 2.
> > - After making necessary improvements, I will send the next patch with
> >   more confidence in the executor correctness.
>
> Ok, I have merged your patches with my patches to create v40
> patches.
>
> My patches are to fix issue with RPR + IGNORE NULLS case. I modified
> ignorenulls_getfuncarginframe() which is called from window functions
> when IGNORE NULLS option is specified. To cope with RPR, I added
> row_is_in_reduced_frame() call to it. Also test cases were added.
>
> Attached are the v40 patches.
>
> Best regards,
> --
> Tatsuo Ishii
> SRA OSS K.K.
> English: http://www.sraoss.co.jp/index_en/
> Japanese:http://www.sraoss.co.jp
>

  171. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-19T06:45:10Z

    While looking into EXPLAIN VERBOSE ANALYZE output of a RPR defined
query, I noticed that the "Output" row of the explain command includes
columns from DEFINE clause (price). This is because columns referenced
in the DEFINE clause must appear on the target list. It is the same
situation as a target list which does not include a column used by
ORDER BY clause like "SELECT price FROM stock ORDER by company".  See
the discussion:
https://www.postgresql.org/message-id/13494.1250901451%40sss.pgh.pa.us

So I think it's ok for now. Opinions?

explain analyze verbose
SELECT company, tdate, price, count(*) OVER w
 FROM stock
 WINDOW w AS (
 PARTITION BY company
 ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
 INITIAL
 PATTERN (A{,2} )
 DEFINE
  A AS price = 200 OR price = 140,
  B AS price = 150
);
                                                        QUERY PLAN                                                         
---------------------------------------------------------------------------------------------------------------------------
 WindowAgg  (cost=83.46..113.37 rows=1200 width=50) (actual time=0.017..0.031 rows=10.00 loops=1)
   Output: company, tdate, price, count(*) OVER w, ((price = 200) OR (price = 140)), (price = 150)
   Window: w AS (PARTITION BY stock.company ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
   Pattern: a{0,2}
   Storage: Memory  Maximum Storage: 17kB
   NFA States: 3 peak, 13 total, 0 merged
   NFA Contexts: 3 peak, 11 total, 8 pruned
   NFA: 2 matched (len 1/1/1), 0 mismatched
   Buffers: shared hit=1
   ->  Sort  (cost=83.37..86.37 rows=1200 width=40) (actual time=0.008..0.009 rows=10.00 loops=1)
         Output: company, tdate, price
         Sort Key: stock.company
         Sort Method: quicksort  Memory: 25kB
         Buffers: shared hit=1
         ->  Seq Scan on public.stock  (cost=0.00..22.00 rows=1200 width=40) (actual time=0.003..0.003 rows=10.00 loops=1)
               Output: company, tdate, price
               Buffers: shared hit=1
 Planning Time: 0.023 ms
 Execution Time: 0.050 ms
(19 rows)

  172. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-19T13:48:20Z

    Hi Ishii-san,

I'm sending a patch that refactors the NFA context absorption logic
for Row Pattern Recognition (RPR).

== Overview ==

This patch fixes a flawed NFA execution model where interleaved
match/advance
logic prevented correct context absorption. The new design reorganizes
execution into a clear three-phase cycle: "match → absorb → advance",
enabling proper state synchronization for absorption.

== Major Changes ==

1. NFA Execution Flow Redesign (nodeWindowAgg.c)

   Previously, match and advance logic were interleaved in
nfa_step_single().
   This interleaved approach prevented proper state synchronization needed
   for absorption - states would scatter to different positions before
   absorption could compare them. Now they are clearly separated:

   - nfa_match(): Convergence phase. Evaluates all waiting states against
     the current row. Checks VAR elements' DEFINE conditions and removes
     failed states.

   - nfa_absorb_contexts(): Absorbs redundant contexts. Performs absorption
     checks at synchronized points after matching completes.

   - nfa_advance(): Divergence phase. Expands epsilon transitions.
     Handles ALT (alternation), END (group termination), and optional
     elements.

   Advantages of three-phase design:
     - Correct absorption: All states evaluate the same row first, ensuring
       synchronized positions for meaningful absorption comparison.
     - Clear semantics: Each phase has single responsibility - convergence,
       optimization, divergence - making the logic easier to understand.
     - Predictable state positions: After match phase, all states are at
       known positions, enabling reliable absorption decisions.

2. Two-flag Absorption Design (rpr.c)

   At planning time, analyzes absorption eligibility and sets two flags:

   - RPR_ELEM_ABSORBABLE: Absorption judgment point (WHERE to compare).
     Set on the VAR element itself for simple unbounded VAR (A+),
     or on END only for groups ((A B)+).

   - RPR_ELEM_ABSORBABLE_BRANCH: Marks the absorbable region.
     Set on ALL elements within the region. Used at runtime to track
     state.isAbsorbable.

   Example - Pattern "(A B)+ C":
     Element 0 (A): ABSORBABLE_BRANCH
     Element 1 (B): ABSORBABLE_BRANCH
     Element 2 (END): ABSORBABLE | ABSORBABLE_BRANCH  <- judgment point
     Element 3 (C): (none)

   Advantages of two-flag design:
     - Correctness: Ensures absorption comparison happens only at
synchronized
       points (e.g., END for groups), not at arbitrary positions.
     - Efficiency: All analysis done at planning time; runtime uses simple
       O(1) flag checks instead of pattern structure analysis.
     - Simplicity: Runtime logic is just monotonic flag propagation -
       once state leaves absorbable region, it stays non-absorbable.

3. Context-level Absorption Flags (execnodes.h)

   Added two flags to RPRNFAContext structure:

   - hasAbsorbableState: True if at least one state is absorbable.
     Indicates whether this context can absorb other contexts.

   - allStatesAbsorbable: True if ALL states are absorbable.
     Indicates whether this context can be absorbed.

   Absorption condition: Ctx1.hasAbsorbableState && Ctx2.allStatesAbsorbable

   Advantages of asymmetric two-flag design:
     - Fast pre-filter: O(1) flag check eliminates most context pairs before
       expensive state-by-state comparison.
     - Flexible absorption: Ctx1 can have extra non-absorbable states and
       still absorb Ctx2, as long as Ctx1 covers all of Ctx2's absorbable
states.
     - Optimized updates: hasAbsorbableState is monotonic (true→false only),
       skipping recalculation once false. allStatesAbsorbable can recover
       (false→true) when non-absorbable states die.

4. State-level isAbsorbable Flag (execnodes.h)

   Added isAbsorbable field to RPRNFAState structure:

   - Computed at creation: sourceAbsorbable && (elem->flags &
ABSORBABLE_BRANCH)
   - Monotonic property: once false, stays false forever
   - Cannot re-enter the absorbable region once left

5. Advance Function Separation

   Split the monolithic processing into per-element-type functions:

   - nfa_advance_alt(): Expands ALT branches. Processes all branches in
     DFS order to preserve lexical priority.

   - nfa_advance_end(): END group repetition logic. Decides loop/exit
     based on count vs min/max constraints.

   - nfa_advance_var(): VAR repetition/exit transitions.

   - nfa_route_to_elem(): Routes to next element. If VAR, adds to waiting
     states; otherwise, recursively expands epsilon transitions.

6. VAR→END Inline Transition

   When a simple VAR (min=max=1) matches and the next element is END,
   we transition to END immediately in the match phase. This is necessary
   for correct absorption: states must be at END to be marked absorbable
   before the absorption check occurs.

7. EXPLAIN Output Improvements (explain.c)

   Changed average length statistics from integer to float with 1 decimal:
   - ExplainPropertyInteger → ExplainPropertyFloat
   - "%.0f" → "%.1f"

== Performance Impact ==

For patterns like "A+ B", context absorption achieves O(n²) → O(n)
complexity improvement.

Example: Input A A A A A B
  - Row 0: Ctx1 at A (count=1)
  - Row 1: Ctx1 at A (count=2), Ctx2 at A (count=1)
    → Ctx1.count >= Ctx2.count, so Ctx2 is absorbed
  - Row 2: Ctx1 at A (count=3), Ctx3 at A (count=1)
    → Ctx3 is absorbed
  - ...

Only one context is maintained throughout, achieving O(n).

== Algorithm Details ==

The absorption algorithm works as follows:

For each pair (older Ctx1, newer Ctx2):
  1. Pre-check: Ctx1.hasAbsorbableState && Ctx2.allStatesAbsorbable
     → If false, skip (fast filter)
  2. Coverage check: For each Ctx2 state with isAbsorbable=true,
     find Ctx1 state with same elemIdx and count >= Ctx2.count
  3. If all Ctx2 absorbable states are covered, absorb Ctx2

The asymmetric design (Ctx1 needs hasAbsorbable, Ctx2 needs allAbsorbable)
allows absorption even when Ctx1 has extra non-absorbable states.

== Key Design Decisions ==

- VAR→END transition in match phase: When a simple VAR (max=1) matches
  and the next element is END, we transition immediately in the match
  phase rather than waiting for advance. This ensures states are at END
  (marked absorbable) before the absorption check.

- Optional VAR skip paths: When advance lands on a VAR with min=0,
  we create both a waiting state AND a skip state (like ALT branches).
  This ensures patterns like "A B? C" work correctly.

- END→END count increment: When transitioning from one END to another
  END within advance, we must increment the outer END's count. This
  handles nested groups like "((A|B)+)+" correctly.

- initialAdvance flag: The first advance after context creation must
  skip FIN recording. Reaching FIN without evaluating any VAR would
  create a zero-length match, which is invalid.

== Future Work ==

- Reluctant quantifiers: Currently rejected at parse time with error
  "reluctant quantifiers are not yet supported". The semantics for mixed
  patterns (e.g., A+? B+) and the optimal implementation approach need
  further clarification.

- State pruning: May be required for reluctant quantifiers (enables early
  match finalization) and context absorption (increases absorption
  eligibility by removing competing states).

- No more large-scale refactoring is planned. Future work will focus on
  quality verification through additional test cases, comprehensive test
  coverage, and memory leak checks.

Please review.

Best regards,
Henson

  173. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-19T23:33:53Z

    Hi Henson,

> Hi Ishii-san,
> 
> I'm sending a patch that refactors the NFA context absorption logic
> for Row Pattern Recognition (RPR).

Thanks for the new patches. I will create v41 patch.

> - No more large-scale refactoring is planned. Future work will focus on
>   quality verification through additional test cases, comprehensive test
>   coverage, and memory leak checks.

Maybe time to run pgindent? I can run pgindent before creating v41.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  174. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-19T23:35:14Z

    Fix references to external documentation not included

2026년 1월 20일 (화) AM 8:34, Tatsuo Ishii <ishii@postgresql.org>님이 작성:

> Hi Henson,
>
> > Hi Ishii-san,
> >
> > I'm sending a patch that refactors the NFA context absorption logic
> > for Row Pattern Recognition (RPR).
>
> Thanks for the new patches. I will create v41 patch.
>
> > - No more large-scale refactoring is planned. Future work will focus on
> >   quality verification through additional test cases, comprehensive test
> >   coverage, and memory leak checks.
>
> Maybe time to run pgindent? I can run pgindent before creating v41.
>
> Best regards,
> --
> Tatsuo Ishii
> SRA OSS K.K.
> English: http://www.sraoss.co.jp/index_en/
> Japanese:http://www.sraoss.co.jp
>

  175. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-19T23:39:57Z

    Hi Ishii-san,

> - No more large-scale refactoring is planned. Future work will focus on
> >   quality verification through additional test cases, comprehensive test
> >   coverage, and memory leak checks.
>
> Maybe time to run pgindent? I can run pgindent before creating v41.
>

Thanks! I'm tied up with work for a few days. Could you please run
pgindent when you create v41? I'll pick it up from cfbot next week.
I'd appreciate it.

Best regards,
Henson

  176. Re: Row pattern recognition

    Michael Paquier <michael@paquier.xyz> — 2026-01-19T23:40:43Z

    On Mon, Jan 19, 2026 at 12:32:40AM +0900, Henson Choi wrote:
> I'm writing to document the detailed design of context absorption
> optimization in Row Pattern Recognition (RPR). This is mainly for
> future reference - honestly, I'm afraid I'll forget these details
> myself in a few months, and I'd rather have them written down
> somewhere public than buried in my notes. I don't want to repeat the
> mistake of famous mathematical theorems
> where brilliant ideas were lost because "the margin was too narrow
> to contain the proof" (yes, I'm looking at you, Fermat!). I certainly
> don't want future PostgreSQL hackers to spend the next 358 years
> trying to re-implement context absorption because the design rationale
> was lost. Fortunately, the PostgreSQL community has provided us with
> sufficient digital margins. :)

Note: this message is clearly AI-generated to me, and these are
usually super sloppy.  (I'd suggest to stop posting semi-generated
messages as they tend to be made of mostly unhelpful contents as they
lack judgement and context, but there's no rule against that AFAIK
around here, as well.)
--
Michael

  177. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-19T23:51:30Z

    Hi Michael,

2026년 1월 20일 (화) AM 8:40, Michael Paquier <michael@paquier.xyz>님이 작성:

> On Mon, Jan 19, 2026 at 12:32:40AM +0900, Henson Choi wrote:
> > I'm writing to document the detailed design of context absorption
> > optimization in Row Pattern Recognition (RPR). This is mainly for
> > future reference - honestly, I'm afraid I'll forget these details
> > myself in a few months, and I'd rather have them written down
> > somewhere public than buried in my notes. I don't want to repeat the
> > mistake of famous mathematical theorems
> > where brilliant ideas were lost because "the margin was too narrow
> > to contain the proof" (yes, I'm looking at you, Fermat!). I certainly
> > don't want future PostgreSQL hackers to spend the next 358 years
> > trying to re-implement context absorption because the design rationale
> > was lost. Fortunately, the PostgreSQL community has provided us with
> > sufficient digital margins. :)
>
> Note: this message is clearly AI-generated to me, and these are
> usually super sloppy.  (I'd suggest to stop posting semi-generated
> messages as they tend to be made of mostly unhelpful contents as they
> lack judgement and context, but there's no rule against that AFAIK
> around here, as well.)
>

Fair point. I can read technical documentation but lack
English communication skills, so I rely on AI for
translation. Actually, AI is what enabled me to start
contributing to PostgreSQL in the first place. I'll try
to make future posts more readable.


> --
> Michael
>

Thanks for the feedback.

Best regards,
Henson

  178. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-20T00:37:04Z

    >> Note: this message is clearly AI-generated to me, and these are
>> usually super sloppy.  (I'd suggest to stop posting semi-generated
>> messages as they tend to be made of mostly unhelpful contents as they
>> lack judgement and context, but there's no rule against that AFAIK
>> around here, as well.)
>>
> 
> Fair point. I can read technical documentation but lack
> English communication skills, so I rely on AI for
> translation. Actually, AI is what enabled me to start
> contributing to PostgreSQL in the first place. I'll try
> to make future posts more readable.

Using AI for translation is no problem for me. In fact, since I am not
good at English, sometimes I am googling to look for good translation
of a sentence in my message.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  179. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-20T00:39:03Z

    >> Maybe time to run pgindent? I can run pgindent before creating v41.
>>
> 
> Thanks! I'm tied up with work for a few days. Could you please run
> pgindent when you create v41? I'll pick it up from cfbot next week.
> I'd appreciate it.

Sure no problem.
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  180. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-20T00:47:25Z

    Hi Ishii-san,

I agree - this is expected behavior.

DEFINE expressions must be in the target list - this is a
prerequisite for NFA pattern matching. They go through the
standard expression pipeline:

2026년 1월 19일 (월) PM 3:45, Tatsuo Ishii <ishii@postgresql.org>님이 작성:

> While looking into EXPLAIN VERBOSE ANALYZE output of a RPR defined
> query, I noticed that the "Output" row of the explain command includes
> columns from DEFINE clause (price). This is because columns referenced
> in the DEFINE clause must appear on the target list. It is the same
> situation as a target list which does not include a column used by
> ORDER BY clause like "SELECT price FROM stock ORDER by company".  See
> the discussion:
> https://www.postgresql.org/message-id/13494.1250901451%40sss.pgh.pa.us
>
> So I think it's ok for now. Opinions?
>
> explain analyze verbose
> SELECT company, tdate, price, count(*) OVER w
>  FROM stock
>  WINDOW w AS (
>  PARTITION BY company
>  ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>  INITIAL
>  PATTERN (A{,2} )
>  DEFINE
>   A AS price = 200 OR price = 140,
>   B AS price = 150
> );
>                                                         QUERY PLAN
>
>
> ---------------------------------------------------------------------------------------------------------------------------
>  WindowAgg  (cost=83.46..113.37 rows=1200 width=50) (actual
> time=0.017..0.031 rows=10.00 loops=1)
>    Output: company, tdate, price, count(*) OVER w, ((price = 200) OR
> (price = 140)), (price = 150)
>    Window: w AS (PARTITION BY stock.company ROWS BETWEEN CURRENT ROW AND
> UNBOUNDED FOLLOWING)
>    Pattern: a{0,2}
>    Storage: Memory  Maximum Storage: 17kB
>    NFA States: 3 peak, 13 total, 0 merged
>    NFA Contexts: 3 peak, 11 total, 8 pruned
>    NFA: 2 matched (len 1/1/1), 0 mismatched
>    Buffers: shared hit=1
>    ->  Sort  (cost=83.37..86.37 rows=1200 width=40) (actual
> time=0.008..0.009 rows=10.00 loops=1)
>          Output: company, tdate, price
>          Sort Key: stock.company
>          Sort Method: quicksort  Memory: 25kB
>          Buffers: shared hit=1
>          ->  Seq Scan on public.stock  (cost=0.00..22.00 rows=1200
> width=40) (actual time=0.003..0.003 rows=10.00 loops=1)
>                Output: company, tdate, price
>                Buffers: shared hit=1
>  Planning Time: 0.023 ms
>  Execution Time: 0.050 ms
> (19 rows)
>

SQL: DEFINE A AS price = 200

    [PARSE]
    ResTarget { name: "A", val: "price = 200" }
        ↓ findTargetlistEntrySQL99()
    Added to query targetlist
        ↓
    [PLAN]
    WindowAgg.defineClause = List<TargetEntry>
        ↓
    [RUNTIME]
    ExecEvalExpr() called for EACH ROW
        ↓
    varMatched[A] = true/false → used by NFA pattern matching

This is the same as ORDER BY columns not in SELECT - they must
be in the internal target list for execution, as Tom Lane
explained in the thread you referenced.

Best regards,
Henson

  181. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-20T02:29:34Z

    Hi Ishii-san,

While reviewing the code, I noticed a potential improvement
in 'register_reduced_frame_map()':

diff --git a/src/backend/executor/nodeWindowAgg.c
b/src/backend/executor/nodeWindowAgg.c
index 5185ad40237..3a71c279945 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -4770,7 +4770,7 @@ register_reduced_frame_map(WindowAggState *winstate,
int64 pos, int val)
        if (pos < 0)
                elog(ERROR, "wrong pos: " INT64_FORMAT, pos);

-       if (pos > winstate->alloc_sz - 1)
+       while (pos > winstate->alloc_sz - 1)
        {
                realloc_sz = winstate->alloc_sz * 2;

In practice, 'pos' is always sequential (0, 1, 2, ...),
so the current 'if' works fine - doubling once is always
sufficient for +1 increment.

However, using 'while' would be more defensive, handling
any theoretical case where 'pos' might jump by more than
2x the current size.

It's a trivial change with no functional impact on normal
operation, but makes the code more robust.

What do you think?

Best regards

  182. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-20T03:14:45Z

    > Hi Ishii-san,
> 
> While reviewing the code, I noticed a potential improvement
> in 'register_reduced_frame_map()':
> 
> diff --git a/src/backend/executor/nodeWindowAgg.c
> b/src/backend/executor/nodeWindowAgg.c
> index 5185ad40237..3a71c279945 100644
> --- a/src/backend/executor/nodeWindowAgg.c
> +++ b/src/backend/executor/nodeWindowAgg.c
> @@ -4770,7 +4770,7 @@ register_reduced_frame_map(WindowAggState *winstate,
> int64 pos, int val)
>         if (pos < 0)
>                 elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
> 
> -       if (pos > winstate->alloc_sz - 1)
> +       while (pos > winstate->alloc_sz - 1)
>         {
>                 realloc_sz = winstate->alloc_sz * 2;
> 
> In practice, 'pos' is always sequential (0, 1, 2, ...),
> so the current 'if' works fine - doubling once is always
> sufficient for +1 increment.
> 
> However, using 'while' would be more defensive, handling
> any theoretical case where 'pos' might jump by more than
> 2x the current size.
> 
> It's a trivial change with no functional impact on normal
> operation, but makes the code more robust.
> 
> What do you think?

Looks good improvement to me. Will merge into v41.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  183. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-20T03:20:40Z

    > Fix references to external documentation not included

After applying the patch, I got rpr_explain regression test diffs as
below as attached.

EXTRA_TESTS=rpr_explain make check

Can you please review the regression diffs so that I can confirm to
replace the expected file or not?

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  184. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-20T03:29:14Z

    >
> > Fix references to external documentation not included
>
> After applying the patch, I got rpr_explain regression test diffs as
> below as attached.
>
> EXTRA_TESTS=rpr_explain make check
>
> Can you please review the regression diffs so that I can confirm to
> replace the expected file or not?


Attached the missing expected file. Please replace.

  185. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-01-20T10:28:34Z

    Hi Henson,

>>> Maybe time to run pgindent? I can run pgindent before creating v41.
>>>
>> 
>> Thanks! I'm tied up with work for a few days. Could you please run
>> pgindent when you create v41? I'll pick it up from cfbot next week.
>> I'd appreciate it.
> 
> Sure no problem.

Pgindent done. Attached are the v41 patches.  Note that I have updated
typedefs.list because some RPR related typedefs were missed.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  186. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-01-21T03:27:15Z

    Hi hackers,

I'd like to propose an extension to the Row Pattern Recognition (RPR)
absorption mechanism to handle anchored patterns (those starting with
START or similar prefix elements).

== Background ==

The current RPR implementation includes an absorption optimization that
reduces time complexity from O(n²) to O(n) for unbounded quantifiers
like A+. When multiple match contexts are at the same ABSORBABLE element,
the context that started earlier (having matched more rows) absorbs
later-starting contexts, eliminating redundant processing.

For example, with pattern "A+ B":
  - Ctx1 starts at row 0, matching A
  - Ctx2 starts at row 1, matching A
  - Ctx1 always has more A's → Ctx2 is absorbed (removed)
  - Result: Only 1 context maintained, O(n) complexity

== The Problem ==

This absorption mechanism breaks down for anchored patterns like
"START SECOND A+ B". The PREFIX elements (START, SECOND) block
absorption because contexts at different PREFIX positions cannot
be meaningfully compared.

Without absorption, we regress to O(n²) behavior as contexts
accumulate without being eliminated.

== Proposed Solution: Original/Alternate Simultaneous Progression ==

The key insight is to track two logical "paths" within each context:

1. Original: Starts at Element 0 (PREFIX + BODY)
   - can_absorb = true
   - Can be used as matchedState (actual match result)

2. Alternate: Starts at Element 2 (BODY only, skipping PREFIX)
   - can_be_absorbed = true
   - Used only for absorption eligibility checking
   - Cannot be used as matchedState

Both paths process the same row data simultaneously. The alternate
path serves as a "shadow" that indicates whether the context is
eligible for absorption.

== Absorption Conditions ==

For Ctx1 to absorb Ctx2, all conditions must be met:

1. Ctx1.can_absorb = true (original not outside pattern region)
2. Ctx2.can_be_absorbed = true (alternate still alive in BODY)
3. Ctx2's original has reached ABSORBABLE (PREFIX matching confirmed)
4. Ctx1 started before Ctx2

Note: No count comparison is needed. The alternate being alive
confirms absorption eligibility; the earlier-starting context
always wins.

== Why Wait for Ctx2's Original to Reach ABSORBABLE? ==

We cannot absorb Ctx2 immediately when it starts. We must wait
until Ctx2's original successfully passes through PREFIX elements
(START, SECOND) and reaches the ABSORBABLE point (A+). This
confirms that Ctx2's PREFIX matching succeeded before we remove it.

== Why Absorption is Safe (Common Fate) ==

Once both contexts reach the same ABSORBABLE element (A+), they
share a "common fate":

  - Same element position, same future data stream
  - Same choices ahead (continue A+ or transition to B)
  - If Ctx1 succeeds → Ctx2 would have succeeded too (but shorter match)
  - If Ctx1 fails → Ctx2 would have failed too (nothing lost)
  - The only difference is in the past (rows already matched),
    which doesn't affect future matching decisions

This property guarantees that absorption is safe: we never lose
a potential match by absorbing a later-starting context into an
earlier one.

== Flag Design ==

Compile-time (Element flags):
  - RPR_ELEM_ABSORBABLE: Unchanged (marks WHERE comparison happens)
  - RPR_ELEM_ABSORBABLE_BRANCH_PREFIX: New (PREFIX region, blocks
absorption)
  - RPR_ELEM_ABSORBABLE_BRANCH_BODY: New (BODY region, allows absorption)

Runtime (State variables):
  - can_absorb: "Can absorb others" (original=true, alternate=false)
  - can_be_absorbed: "Can be absorbed" (PREFIX→false, BODY→true)

Both flags are monotonic: once false, they cannot become true again.

== Complexity Analysis ==

With this extension:
  - Maximum concurrent contexts: PREFIX_length + 1
  - For "START SECOND A+ B": max 3 contexts at any time
  - Overall complexity: O(n) maintained

Example flow for "START SECOND A+ B" with data r0-r4(A,A,A,A,B):
  Row 0: Ctx1 starts                                           [1]
  Row 1: Ctx2 starts (cannot absorb: PREFIX not confirmed)     [2]
  Row 2: Ctx3 starts                                           [3]
  Row 3: Ctx4 starts, Ctx2 absorbed (PREFIX confirmed)         [3]
  Row 4: Ctx1 matches B, Ctx3-5 discarded                      [1]

Note: This is not an immediate priority. Stabilizing the current
RPR patch comes first. I'm sharing this design early to get feedback
and ensure the approach is sound before implementation.

I have a working design document with detailed state transition
rules and examples. Happy to share more details.

Thoughts?

--
Best regards,
Henson

  187. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-02T07:20:00Z

    The series of patches are to implement the row pattern recognition
(SQL/RPR) feature. Currently the implementation is a subset of SQL/RPR
(ISO/IEC 19075-2:2016). Namely, implementation of some features of
R020 (WINDOW clause). R010 (MATCH_RECOGNIZE) is out of the scope of
the patches.

Attached are the v42 patches.
The difference from v41 include:

- Add Peter as a reviewer to commit message. He gave a good valuable
  suggestions to the gram.y patch. I accidentally forgot to credit his
  name.

- Tons of enhancements from Henson.

-- Fully rewrite absorption logic (optimizer/plan/rpr.c).
   Example: (a+ b* c?){2,} -> (a+" b* c?){2,}

-- Add safety check to tryMultiplyQuantifiers (optimizer/plan/rpr.c)
   Example: (A{2}){2,3} cannot be rewritten to A{4,6} because it allows
   5 times reputation which is not correct. Perform multiplication
   optimization only in safe case.

-- Fix infinite loop/segfault when unlimited quantifiers (e.g. (A*)*,
   (A+)+) are used.

-- Optimize unlimited quantifiers if possible (e.g. (A*)* -> A*, (A+)+ -> A+)

-- Fix segfault case (SELECT id, flag ...WINDOW w AS (... DEFINE T AS flag)

-- More accurate error position report

-- Some code refactoring in parser/planner patches

-- Split regression test file into rpr.sql and rpr_base.sql

-- Fix outfuncs.c and readfuncs.c

-- Fix some corner cases bug of regular expression optimization

-- Remove unused PATTERN variable to enhance performance

-- Change treatment of undefined DEFINE variables. Previously if
   variable "A" is used in PATTERN but not defined in DEFINE, A is
   automatically defined as "A AS TRUE". Now A is evaluated as TRUE in
   executor and the generated entry for A is removed. This improves
   memory usage.

-- EXPLAIN now shows the absorbable points (a+")

- Bug fix from me.

-- Assertion failure when two or more window clauses are used.
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  188. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-03T00:00:39Z

    Hi Henson,

> The series of patches are to implement the row pattern recognition
> (SQL/RPR) feature. Currently the implementation is a subset of SQL/RPR
> (ISO/IEC 19075-2:2016). Namely, implementation of some features of
> R020 (WINDOW clause). R010 (MATCH_RECOGNIZE) is out of the scope of
> the patches.
> 
> Attached are the v42 patches.

I found a small typo in a function comment: non-ASCII letter was
used. In my understanding, using non-ASCII letters in the program
files are not recommended. Patch attached.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  189. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-03T08:40:01Z

    Hi Tatsuo,

I found a small typo in a function comment: non-ASCII letter was
> used. In my understanding, using non-ASCII letters in the program
> files are not recommended. Patch attached.
>

Thank you for catching this! You are absolutely right. I apologize for
using non-ASCII characters in the code comments.

I've reviewed all the RPR-related files and found additional instances
of non-ASCII characters:
- Arrows "→" instead of "->"
- Superscript "²" instead of "^2"
- Greater-than-or-equal "≥" instead of ">="

Found in:
- src/backend/optimizer/plan/rpr.c (1 instance)
- src/backend/executor/nodeWindowAgg.c (5 instances)
- src/include/nodes/execnodes.h (4 instances)
- src/test/regress/sql/rpr.sql (7 instances)
- src/test/regress/sql/rpr_base.sql (15 instances)
- src/test/regress/expected/rpr.out (7 instances)
- src/test/regress/expected/rpr_base.out (15 instances)

Total: 54 instances across 7 files.

To ensure comprehensive coverage, I created a Python verification tool
that checks for non-ASCII characters and reports their Unicode code
points. I've verified all 46 files changed in the v42 patch series,
and confirmed no additional non-ASCII characters remain.

I'll be more careful going forward to use only ASCII characters in
source files and comments. Updated patch attached.

Best regards,
Henson

  190. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-04T00:04:51Z

    > Thank you for catching this! You are absolutely right. I apologize for
> using non-ASCII characters in the code comments.
> 
> I've reviewed all the RPR-related files and found additional instances
> of non-ASCII characters:
> - Arrows "→" instead of "->"
> - Superscript "²" instead of "^2"
> - Greater-than-or-equal "≥" instead of ">="
> 
> Found in:
> - src/backend/optimizer/plan/rpr.c (1 instance)
> - src/backend/executor/nodeWindowAgg.c (5 instances)
> - src/include/nodes/execnodes.h (4 instances)
> - src/test/regress/sql/rpr.sql (7 instances)
> - src/test/regress/sql/rpr_base.sql (15 instances)
> - src/test/regress/expected/rpr.out (7 instances)
> - src/test/regress/expected/rpr_base.out (15 instances)
> 
> Total: 54 instances across 7 files.
> 
> To ensure comprehensive coverage, I created a Python verification tool
> that checks for non-ASCII characters and reports their Unicode code
> points. I've verified all 46 files changed in the v42 patch series,
> and confirmed no additional non-ASCII characters remain.
> 
> I'll be more careful going forward to use only ASCII characters in
> source files and comments. Updated patch attached.

Thank you for the prompt response. Your patch applied cleanly here.
BTW, I have a question regarding NFA related objects initialization in
nodeWindowAgg.c.

In release_partition():
	/* Reset NFA state for new partition */
	winstate->nfaContext = NULL;
	winstate->nfaContextTail = NULL;
	winstate->nfaContextFree = NULL;
	winstate->nfaStateFree = NULL;
	winstate->nfaLastProcessedRow = -1;
	winstate->nfaStatesActive = 0;
	winstate->nfaContextsActive = 0;

While in ExecInitWindowAgg():
	/* Initialize NFA free lists for row pattern matching */
	winstate->nfaContext = NULL;
	winstate->nfaContextTail = NULL;
	winstate->nfaContextFree = NULL;
	winstate->nfaStateFree = NULL;
	winstate->nfaLastProcessedRow = -1;

So the latter does not initialize winstate->nfaStatesActive and
winstate->nfaContextsActive. Can you please explain why?
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  191. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-04T08:54:26Z

    Hi Tatsuo,

BTW, I have a question regarding NFA related objects initialization in
> nodeWindowAgg.c.
>
> In release_partition():
>         /* Reset NFA state for new partition */
>         winstate->nfaContext = NULL;
>         winstate->nfaContextTail = NULL;
>         winstate->nfaContextFree = NULL;
>         winstate->nfaStateFree = NULL;
>         winstate->nfaLastProcessedRow = -1;
>         winstate->nfaStatesActive = 0;
>         winstate->nfaContextsActive = 0;
>
> While in ExecInitWindowAgg():
>         /* Initialize NFA free lists for row pattern matching */
>         winstate->nfaContext = NULL;
>         winstate->nfaContextTail = NULL;
>         winstate->nfaContextFree = NULL;
>         winstate->nfaStateFree = NULL;
>         winstate->nfaLastProcessedRow = -1;
>
> So the latter does not initialize winstate->nfaStatesActive and
> winstate->nfaContextsActive. Can you please explain why?
>

You're absolutely right to notice this inconsistency. Let me explain:

## Why ExecInitWindowAgg() omits these two fields

In ExecInitWindowAgg(), the WindowAggState structure is created via
makeNode(WindowAggState), which internally uses palloc0(). This
zero-initializes all memory, so:

  - All pointers → NULL (0)
  - nfaStatesActive → 0
  - nfaContextsActive → 0
  - All other statistics counters → 0

Technically, only nfaLastProcessedRow = -1 truly needs explicit
initialization since it requires a non-zero value. The nfaContext = NULL
assignments are redundant but follow PostgreSQL's convention of
explicitly setting pointers to NULL for code clarity and documentation.

The pattern in ExecInitWindowAgg() follows PostgreSQL's common practice:
  - Pointers: explicitly initialized to NULL (for readability)
  - Counters: implicitly zero-initialized via palloc0() (intentionally
omitted)

This same pattern appears throughout the function for other fields too.
For example, framehead_slot and frametail_slot are explicitly set to NULL,
while framehead_valid and frametail_valid are left zero-initialized.

## What nfaStatesActive and nfaContextsActive do

These two fields serve as internal counters for tracking active NFA
resources:

nfaStatesActive: Tracks the current number of active NFA states in memory.
  - Incremented in nfa_state_alloc() when allocating a new state
  - Decremented in nfa_state_free() when returning a state to the free list
  - Used to calculate nfaStatesMax (peak usage) for EXPLAIN ANALYZE

nfaContextsActive: Tracks the current number of active NFA contexts.
  - Incremented in nfa_context_alloc() when creating a new context
  - Decremented in nfa_context_free() when recycling a context
  - Used to calculate nfaContextsMax (peak usage) for EXPLAIN ANALYZE

Both counters start at 0 and are updated during NFA execution to track
resource usage for performance monitoring.

## Why release_partition() must explicitly reset them

release_partition() is called when moving to a new partition and must
reset per-partition state. Unlike ExecInitWindowAgg() which creates a
fresh structure, release_partition() reuses an existing structure that
contains values from the previous partition.

The root cause of the reset requirement is that release_partition()
calls MemoryContextReset(winstate->partcontext), which frees all
partition-local memory. This includes:
  - All NFA context structures pointed to by nfaContext
  - All NFA state nodes in the free lists (nfaContextFree, nfaStateFree)
  - All other partition-specific data

As a consequence of this memory reset:
  1. The NFA pointers (nfaContext, nfaContextFree, etc.) must be set to
     NULL to avoid dangling pointers
  2. The active resource counters (nfaStatesActive, nfaContextsActive)
     must be reset to 0 since all the resources they were counting have
     been freed

Example scenario:
  Partition 1 processing:
    - nfaStatesActive increments one-by-one as states are allocated,
      decrements as they are freed, fluctuates during execution,
      ends at some non-zero value (e.g., 3)
    - nfaContextsActive similarly fluctuates, ends at some value (e.g., 2)

  Partition 2 starts → release_partition() called:
    - MemoryContextReset(partcontext) frees all NFA structures
    - nfaContext, nfaContextFree, etc. → must set to NULL
    - nfaStatesActive: 3 → must reset to 0
    - nfaContextsActive: 2 → must reset to 0

These counters track instantaneous active resources within a partition,
so they must be reset to 0 when starting a new partition. In contrast,
cumulative statistics like nfaStatesMax and nfaStatesTotalCreated are
NOT reset because they track overall query-wide statistics across all
partitions.

## Why only these two counters?

You might wonder why I don't propose initializing all the other NFA
statistics fields as well. ExecInitWindowAgg() currently omits explicit
initialization for many statistics fields:

  Per-partition internal counters (reset at partition boundaries):
    - nfaStatesActive = 0;        (omitted)
    - nfaContextsActive = 0;      (omitted)

  Query-wide cumulative statistics (never reset):
    - nfaStatesMax = 0;                 (omitted)
    - nfaStatesTotalCreated = 0;        (omitted)
    - nfaStatesMerged = 0;              (omitted)
    - nfaContextsMax = 0;               (omitted)
    - nfaContextsTotalCreated = 0;      (omitted)
    - nfaContextsAbsorbed = 0;          (omitted)
    - nfaContextsSkipped = 0;           (omitted)
    - nfaContextsPruned = 0;            (omitted)
    - nfaMatchesSucceeded = 0;          (omitted)
    - nfaMatchesFailed = 0;             (omitted)
    - nfaMatchLen = {0, 0, 0};          (omitted)
    - nfaFailLen = {0, 0, 0};           (omitted)
    - nfaAbsorbedLen = {0, 0, 0};       (omitted)
    - nfaSkippedLen = {0, 0, 0};        (omitted)

The key distinction is that nfaStatesActive and nfaContextsActive are
the ONLY statistics that get reset in release_partition(). All the other
statistics are cumulative across the entire query and should never be
reset between partitions.

For example, nfaStatesMax tracks the peak number of active states across
ALL partitions, not just within one partition. Similarly,
nfaStatesTotalCreated accumulates the total count across all partitions.
These cumulative statistics start at 0 (via palloc0) and only increase,
never getting reset.

Therefore, I believe it only makes sense to add explicit initialization
for nfaStatesActive and nfaContextsActive, since these are the only two
that release_partition() also explicitly resets. Adding explicit
initialization for all the cumulative statistics would be misleading,
as it might suggest they get reset somewhere (which they don't).

## Proposal for consistency

For consistency with release_partition(), I can add just the two
per-partition counter initializations to ExecInitWindowAgg():

  /* Initialize NFA free lists for row pattern matching */
  winstate->nfaContext = NULL;
  winstate->nfaContextTail = NULL;
  winstate->nfaContextFree = NULL;
  winstate->nfaStateFree = NULL;
  winstate->nfaLastProcessedRow = -1;
  winstate->nfaStatesActive = 0;        // Add this
  winstate->nfaContextsActive = 0;      // Add this

This would make both ExecInitWindowAgg() and release_partition()
initialize the same set of per-partition NFA fields, making the code
more uniform and easier to review. The cumulative statistics fields
remain omitted in both functions, which correctly reflects that they
are never reset.

Functionally, this change makes no difference since palloc0() already
zeros these fields, but it improves code consistency and makes the
per-partition vs. query-wide distinction clearer.

Would you like me to include this change in the next patch?

Best regards,
Henson

  192. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-04T10:22:22Z

    Hi Henson,

> Therefore, I believe it only makes sense to add explicit initialization
> for nfaStatesActive and nfaContextsActive, since these are the only two
> that release_partition() also explicitly resets.

That makes sense.

> Adding explicit
> initialization for all the cumulative statistics would be misleading,
> as it might suggest they get reset somewhere (which they don't).
> 
> ## Proposal for consistency
> 
> For consistency with release_partition(), I can add just the two
> per-partition counter initializations to ExecInitWindowAgg():
> 
>   /* Initialize NFA free lists for row pattern matching */
>   winstate->nfaContext = NULL;
>   winstate->nfaContextTail = NULL;
>   winstate->nfaContextFree = NULL;
>   winstate->nfaStateFree = NULL;
>   winstate->nfaLastProcessedRow = -1;
>   winstate->nfaStatesActive = 0;        // Add this
>   winstate->nfaContextsActive = 0;      // Add this
> 
> This would make both ExecInitWindowAgg() and release_partition()
> initialize the same set of per-partition NFA fields, making the code
> more uniform and easier to review. The cumulative statistics fields
> remain omitted in both functions, which correctly reflects that they
> are never reset.
> 
> Functionally, this change makes no difference since palloc0() already
> zeros these fields, but it improves code consistency and makes the
> per-partition vs. query-wide distinction clearer.
> 
> Would you like me to include this change in the next patch?

Yes, please.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  193. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-05T00:46:06Z

    Hi Tatsuo,

> Therefore, I believe it only makes sense to add explicit initialization
> > for nfaStatesActive and nfaContextsActive, since these are the only two
> > that release_partition() also explicitly resets.
>
> That makes sense.
>

Thank you for confirming.

> ## Proposal for consistency
> >
> >   /* Initialize NFA free lists for row pattern matching */
> >   winstate->nfaContext = NULL;
> >   winstate->nfaContextTail = NULL;
> >   winstate->nfaContextFree = NULL;
> >   winstate->nfaStateFree = NULL;
> >   winstate->nfaLastProcessedRow = -1;
> >   winstate->nfaStatesActive = 0;        // Add this
> >   winstate->nfaContextsActive = 0;      // Add this
> >
> > Would you like me to include this change in the next patch?
>
> Yes, please.
>


Done. Please see the attached patch.

Best regards,
Henson

  194. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-05T01:25:19Z

    Hi Henson,

>> ## Proposal for consistency
>> >
>> >   /* Initialize NFA free lists for row pattern matching */
>> >   winstate->nfaContext = NULL;
>> >   winstate->nfaContextTail = NULL;
>> >   winstate->nfaContextFree = NULL;
>> >   winstate->nfaStateFree = NULL;
>> >   winstate->nfaLastProcessedRow = -1;
>> >   winstate->nfaStatesActive = 0;        // Add this
>> >   winstate->nfaContextsActive = 0;      // Add this
>> >
>> > Would you like me to include this change in the next patch?
>>
>> Yes, please.
>>
> 
> 
> Done. Please see the attached patch.

Looks good to me.

BTW, I noticed that following test now succeeds with v42 patch.
In rpr_explain.sql test 11.3:

-- Test 11.3: Consecutive increasing values (using PREV)
-- FIXME: The original pattern was:
--   DEFINE A AS v > PREV(v) OR PREV(v) IS NULL
-- This causes "ERROR: unrecognized node type: 15" (T_FuncExpr) because
-- NullTest(FuncExpr(PREV)) is not properly handled somewhere in the planner.
-- The expression v > PREV(v) works fine, but PREV(v) IS NULL fails.
-- Using COALESCE(PREV(v), 0) as a workaround until the bug is fixed.
EXPLAIN (ANALYZE, BUFFERS OFF, COSTS OFF, TIMING OFF, SUMMARY OFF)
SELECT count(*) OVER w
FROM generate_series(1, 50) AS s(v)
WINDOW w AS (
    ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
    AFTER MATCH SKIP PAST LAST ROW
    PATTERN (A{3,})
    DEFINE A AS v > COALESCE(PREV(v), 0)
);

I changed 
   DEFINE A AS v > COALESCE(PREV(v), 0)
to
   DEFINE A AS v > PREV(v) OR PREV(v) IS NULL
 and get following result.

EXPLAIN (ANALYZE, BUFFERS OFF, COSTS OFF, TIMING OFF, SUMMARY OFF)
SELECT count(*) OVER w
FROM generate_series(1, 50) AS s(v)
WINDOW w AS (
    ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
    AFTER MATCH SKIP PAST LAST ROW
    PATTERN (A{3,})
    DEFINE A AS v > PREV(v) OR PREV(v) IS NULL
);
                              QUERY PLAN                              
----------------------------------------------------------------------
 WindowAgg (actual rows=50.00 loops=1)
   Window: w AS (ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
   Pattern: a{3,}"
   Storage: Memory  Maximum Storage: 18kB
   NFA States: 3 peak, 99 total, 0 merged
   NFA Contexts: 2 peak, 51 total, 0 pruned
   NFA: 1 matched (len 50/50/50.0), 0 mismatched
   NFA: 49 absorbed (len 1/1/1.0), 0 skipped
   ->  Function Scan on generate_series s (actual rows=50.00 loops=1)
(9 rows)

Probably we can restore 11.3 test in v43?

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  195. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-05T14:28:37Z

    Hi Tatsuo,

> Done. Please see the attached patch.
>
> Looks good to me.
>

Thank you for reviewing.


> I changed
>    DEFINE A AS v > COALESCE(PREV(v), 0)
> to
>    DEFINE A AS v > PREV(v) OR PREV(v) IS NULL
>  and get following result.
>
> EXPLAIN (ANALYZE, BUFFERS OFF, COSTS OFF, TIMING OFF, SUMMARY OFF)
> SELECT count(*) OVER w
> FROM generate_series(1, 50) AS s(v)
> WINDOW w AS (
>     ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>     AFTER MATCH SKIP PAST LAST ROW
>     PATTERN (A{3,})
>     DEFINE A AS v > PREV(v) OR PREV(v) IS NULL
> );
>                               QUERY PLAN
> ----------------------------------------------------------------------
>  WindowAgg (actual rows=50.00 loops=1)
>    Window: w AS (ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
>    Pattern: a{3,}"
>    Storage: Memory  Maximum Storage: 18kB
>    NFA States: 3 peak, 99 total, 0 merged
>    NFA Contexts: 2 peak, 51 total, 0 pruned
>    NFA: 1 matched (len 50/50/50.0), 0 mismatched
>    NFA: 49 absorbed (len 1/1/1.0), 0 skipped
>    ->  Function Scan on generate_series s (actual rows=50.00 loops=1)
> (9 rows)
>
> Probably we can restore 11.3 test in v43?
>


Yes, I've already restored Test 11.3 with the original pattern:

  DEFINE A AS v > PREV(v) OR PREV(v) IS NULL

Additionally, I've implemented cross-platform normalization for Storage
memory values
to address the platform differences (18kB vs 19kB, 23kB vs 24kB, etc.)
observed between
macOS and Rocky Linux 10.

The attached patch includes:

1. Test 11.3 restoration with original PREV(v) IS NULL pattern
2. Cross-platform Storage normalization using rpr_explain_filter() PL/pgSQL
function
   - Normalizes text format: "Maximum Storage: 18kB" -> "Maximum Storage:
NkB"
   - Normalizes JSON format: "Maximum Storage": 17 -> "Maximum Storage": 0
   - Normalizes XML format: <Maximum-Storage>17</Maximum-Storage> ->
<Maximum-Storage>0</Maximum-Storage>
   - Preserves NFA statistics unchanged (they are test assertions)
3. All EXPLAIN statements wrapped with rpr_explain_filter()
4. Removed rpr_explain_1.out (no longer needed)
5. Added rpr_explain to parallel_schedule test suite

Patch statistics:
 src/test/regress/expected/rpr_explain.out   |  559 +++---
 src/test/regress/expected/rpr_explain_1.out | 1803 -------------------
 src/test/regress/parallel_schedule          |    2 +-
 src/test/regress/sql/rpr_explain.sql        |  294 ++-
 4 files changed, 521 insertions(+), 2137 deletions(-)
 delete mode 100644 src/test/regress/expected/rpr_explain_1.out

Best regards,
Henson

  196. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-07T04:31:43Z

    > Yes, I've already restored Test 11.3 with the original pattern:
> 
>   DEFINE A AS v > PREV(v) OR PREV(v) IS NULL

Good.

> Additionally, I've implemented cross-platform normalization for Storage
> memory values
> to address the platform differences (18kB vs 19kB, 23kB vs 24kB, etc.)
> observed between
> macOS and Rocky Linux 10.
> 
> The attached patch includes:
> 
> 1. Test 11.3 restoration with original PREV(v) IS NULL pattern
> 2. Cross-platform Storage normalization using rpr_explain_filter() PL/pgSQL
> function

+-- Filter function to normalize Storage memory values only (not NFA statistics)
+-- Works for text, JSON, and XML formats

What about add reasoning why only storage memory values are normalized
something like:

-- NFA statistics should not change between platforms.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  197. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-07T05:41:26Z

    Hi Tatsuo,

+-- Filter function to normalize Storage memory values only (not NFA
> statistics)
> +-- Works for text, JSON, and XML formats
>
> What about add reasoning why only storage memory values are normalized
> something like:
>
> -- NFA statistics should not change between platforms.


Done. I've updated the comment to:

-- Filter function to normalize Storage memory values only (not NFA
statistics).
-- NFA statistics should not change between platforms; if they do, it could
-- indicate issues such as uninitialized memory access.
-- Works for text, JSON, and XML formats.

This round focused primarily on reviewing explain.c.  I believe we're
getting close to the final version now.  Only the nodeWindowAgg.c
code review remains on my side.

For the next submission, would it be helpful if I resend all the
incremental patches I've already sent since v42 together as separate
per-patch files?

I've also made some updates to the documentation (advanced.sgml and
ref/select.sgml).  Could you take a look and let me know if there are
any mistakes or if the changes look good to you?

I've sent several incremental patches on top of v42, and the attached
patch is the next one in that series.  Here are the changes:

Bugs fixed:

- Fix server crash in fillRPRPatternAlt() when an inner ALT is the
  last element of an outer ALT's branch.  Both alternations tried to
  set the 'next' pointer on the same endpoint element, triggering
  Assert(pat->elements[endPos].next == RPR_ELEMIDX_INVALID).  Fix by
  redirecting all elements in the branch that share the old target to
  point to the outer ALT's afterAlt instead.
  (Test: PATTERN (C (A | B) | D) -- inner ALT at end of outer branch)

- Add BEGIN element to fillRPRPatternGroup().  Previously only END was
  emitted for quantified groups, so groups with min=0 had no skip path
  to bypass the group content.  Change to BEGIN/END pairs where
  BEGIN.jump points past END (skip) and END.jump points to the first
  child (loop-back).  Add nfa_advance_begin() to the NFA executor for
  group entry and skip handling.
  (Test: PATTERN (A ((B | C) (D | E))* F?) -- * group matching 0 times)

- Fix deparse output producing wrong parenthesization for nested ALT
  patterns.  The flat-loop approach could not correctly suppress double
  parentheses for single-ALT groups or handle depth transitions around
  alternation separators.  Refactor into mutual recursion:
  deparse_rpr_elements, deparse_rpr_group, deparse_rpr_alt, and
  deparse_rpr_var.

- Fix deparse separator ordering for nested ALT: the ' | ' separator
  was emitted before closing inner parentheses, producing output like
  '(c (a | b | )d)' instead of '(c (a | b) | d)'.  Close parens to
  match separator depth before emitting ' | '.

- Fix missing ALT separator registration in deparse_rpr_alt() when an
  ALT is the first element of an outer ALT's branch.  The code only
  checked elem->next (always -1 for ALT markers) but not elem->jump,
  which carries the outer branch's separator position.

- Fix altEndPositions/altBranchStarts length mismatch caused by the
  'if (*idx > branchStart)' guard that skipped empty branches.  Remove
  the guard so both lists always have matching entries.

- Fix RPRPatternNode.reluctant initialization in gram.y.  ALT, SEQ,
  VAR, and GROUP primary productions used false (0) or left it
  uninitialized (defaulting to 0 from palloc0), but 0 is a valid
  ParseLoc meaning "location at offset 0".  Change all four creation
  sites to use -1 (no location), matching the convention used by
  ParseLoc throughout the parser.

- Fix reluctant quantifier display in both explain.c and ruleutils.c.
  A bare variable with reluctant ? (e.g. A?) was displayed as just 'a'
  since there was no quantifier to attach ? to.  Now emit '{1}?' to
  make the reluctant marker unambiguous.

New optimization:

- Add mergeConsecutiveAlts() to the SEQ optimization pipeline.  After
  GROUP{1,1} unwrap, bare alternations like (A | B) become ALT nodes
  in the SEQ.  This step detects consecutive identical ALT nodes and
  wraps them in a GROUP with the appropriate quantifier, e.g.
  (A | B) (A | B) (A | B) -> (A | B){3}.  Combined with the existing
  mergeGroupPrefixSuffix, patterns like (A | B) (A | B)+ (A | B)
  further reduce to (A | B){3,}.

- Extend tryMultiplyQuantifiers() to fold nested GROUP quantifiers
  (e.g. ((A B)+)* -> (A B)*) in addition to VAR quantifiers.

Other changes:

- Add RPR_VARID_BEGIN (252) to rpr.h for the new control element type.
  Reduce RPR_VARID_MAX from 252 to 251 accordingly and update the
  maximum-variables boundary tests.

- Add RPR_DEPTH_NONE (255) as sentinel for top-level elements that
  have no enclosing group.  Reduce RPR_DEPTH_MAX to 254 accordingly.

- Add BEGIN handling to computeAbsorbabilityRecursive() so that
  absorbability flags propagate correctly through group boundaries.

- Extract show_rpr_nfa_stats() from show_windowagg_info() for NFA
  statistics display.

- Replace defensive NULL check in collectPatternVariables() with
  Assert, since the caller guarantees a non-NULL pattern.

- Pair each EXPLAIN test query in rpr_explain.sql with a
  pg_get_viewdef() check via CREATE TEMP VIEW, verifying that both
  the ruleutils (parse tree) and explain (bytecode) deparse paths
  produce consistent PATTERN output.

- Add comment to rpr_explain_filter() explaining why only Storage
  memory values are normalized: NFA statistics should not change
  between platforms, and variation would indicate issues such as
  uninitialized memory access.

Also add EXPLAIN test cases for nested ALT patterns, consecutive ALT
merging, and ALT prefix/suffix merging.

On a side note, I feel that the automata characteristics of RPR --
state transitions, context management, absorbability, etc. -- tend to
require a much larger volume of test cases than typical RDBMS/SQL
features.  This seems unavoidable given the combinatorial nature of
NFA execution.

Best regards,
Henson

  198. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-08T16:05:10Z

    Hi,

Sorry, I forgot to attach the patch file in my previous email.

2026년 2월 7일 (토) PM 2:41, Henson Choi <assam258@gmail.com>님이 작성:

> Hi Tatsuo,
>
> +-- Filter function to normalize Storage memory values only (not NFA
>> statistics)
>> +-- Works for text, JSON, and XML formats
>>
>> What about add reasoning why only storage memory values are normalized
>> something like:
>>
>> -- NFA statistics should not change between platforms.
>
>
> Done. I've updated the comment to:
>
> -- Filter function to normalize Storage memory values only (not NFA
> statistics).
> -- NFA statistics should not change between platforms; if they do, it could
> -- indicate issues such as uninitialized memory access.
> -- Works for text, JSON, and XML formats.
>
> This round focused primarily on reviewing explain.c.  I believe we're
> getting close to the final version now.  Only the nodeWindowAgg.c
> code review remains on my side.
>
> For the next submission, would it be helpful if I resend all the
> incremental patches I've already sent since v42 together as separate
> per-patch files?
>
> I've also made some updates to the documentation (advanced.sgml and
> ref/select.sgml).  Could you take a look and let me know if there are
> any mistakes or if the changes look good to you?
>
> I've sent several incremental patches on top of v42, and the attached
> patch is the next one in that series.  Here are the changes:
>
> Bugs fixed:
>
> - Fix server crash in fillRPRPatternAlt() when an inner ALT is the
>   last element of an outer ALT's branch.  Both alternations tried to
>   set the 'next' pointer on the same endpoint element, triggering
>   Assert(pat->elements[endPos].next == RPR_ELEMIDX_INVALID).  Fix by
>   redirecting all elements in the branch that share the old target to
>   point to the outer ALT's afterAlt instead.
>   (Test: PATTERN (C (A | B) | D) -- inner ALT at end of outer branch)
>
> - Add BEGIN element to fillRPRPatternGroup().  Previously only END was
>   emitted for quantified groups, so groups with min=0 had no skip path
>   to bypass the group content.  Change to BEGIN/END pairs where
>   BEGIN.jump points past END (skip) and END.jump points to the first
>   child (loop-back).  Add nfa_advance_begin() to the NFA executor for
>   group entry and skip handling.
>   (Test: PATTERN (A ((B | C) (D | E))* F?) -- * group matching 0 times)
>
> - Fix deparse output producing wrong parenthesization for nested ALT
>   patterns.  The flat-loop approach could not correctly suppress double
>   parentheses for single-ALT groups or handle depth transitions around
>   alternation separators.  Refactor into mutual recursion:
>   deparse_rpr_elements, deparse_rpr_group, deparse_rpr_alt, and
>   deparse_rpr_var.
>
> - Fix deparse separator ordering for nested ALT: the ' | ' separator
>   was emitted before closing inner parentheses, producing output like
>   '(c (a | b | )d)' instead of '(c (a | b) | d)'.  Close parens to
>   match separator depth before emitting ' | '.
>
> - Fix missing ALT separator registration in deparse_rpr_alt() when an
>   ALT is the first element of an outer ALT's branch.  The code only
>   checked elem->next (always -1 for ALT markers) but not elem->jump,
>   which carries the outer branch's separator position.
>
> - Fix altEndPositions/altBranchStarts length mismatch caused by the
>   'if (*idx > branchStart)' guard that skipped empty branches.  Remove
>   the guard so both lists always have matching entries.
>
> - Fix RPRPatternNode.reluctant initialization in gram.y.  ALT, SEQ,
>   VAR, and GROUP primary productions used false (0) or left it
>   uninitialized (defaulting to 0 from palloc0), but 0 is a valid
>   ParseLoc meaning "location at offset 0".  Change all four creation
>   sites to use -1 (no location), matching the convention used by
>   ParseLoc throughout the parser.
>
> - Fix reluctant quantifier display in both explain.c and ruleutils.c.
>   A bare variable with reluctant ? (e.g. A?) was displayed as just 'a'
>   since there was no quantifier to attach ? to.  Now emit '{1}?' to
>   make the reluctant marker unambiguous.
>
> New optimization:
>
> - Add mergeConsecutiveAlts() to the SEQ optimization pipeline.  After
>   GROUP{1,1} unwrap, bare alternations like (A | B) become ALT nodes
>   in the SEQ.  This step detects consecutive identical ALT nodes and
>   wraps them in a GROUP with the appropriate quantifier, e.g.
>   (A | B) (A | B) (A | B) -> (A | B){3}.  Combined with the existing
>   mergeGroupPrefixSuffix, patterns like (A | B) (A | B)+ (A | B)
>   further reduce to (A | B){3,}.
>
> - Extend tryMultiplyQuantifiers() to fold nested GROUP quantifiers
>   (e.g. ((A B)+)* -> (A B)*) in addition to VAR quantifiers.
>
> Other changes:
>
> - Add RPR_VARID_BEGIN (252) to rpr.h for the new control element type.
>   Reduce RPR_VARID_MAX from 252 to 251 accordingly and update the
>   maximum-variables boundary tests.
>
> - Add RPR_DEPTH_NONE (255) as sentinel for top-level elements that
>   have no enclosing group.  Reduce RPR_DEPTH_MAX to 254 accordingly.
>
> - Add BEGIN handling to computeAbsorbabilityRecursive() so that
>   absorbability flags propagate correctly through group boundaries.
>
> - Extract show_rpr_nfa_stats() from show_windowagg_info() for NFA
>   statistics display.
>
> - Replace defensive NULL check in collectPatternVariables() with
>   Assert, since the caller guarantees a non-NULL pattern.
>
> - Pair each EXPLAIN test query in rpr_explain.sql with a
>   pg_get_viewdef() check via CREATE TEMP VIEW, verifying that both
>   the ruleutils (parse tree) and explain (bytecode) deparse paths
>   produce consistent PATTERN output.
>
> - Add comment to rpr_explain_filter() explaining why only Storage
>   memory values are normalized: NFA statistics should not change
>   between platforms, and variation would indicate issues such as
>   uninitialized memory access.
>
> Also add EXPLAIN test cases for nested ALT patterns, consecutive ALT
> merging, and ALT prefix/suffix merging.
>
> On a side note, I feel that the automata characteristics of RPR --
> state transitions, context management, absorbability, etc. -- tend to
> require a much larger volume of test cases than typical RDBMS/SQL
> features.  This seems unavoidable given the combinatorial nature of
> NFA execution.
>
> Best regards,
> Henson
>

  199. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-09T07:05:20Z

    Hi Henson,

> Sorry, I forgot to attach the patch file in my previous email.

Thanks for the patch!  Patch cleanly applied and regression test
passed here.

>> On a side note, I feel that the automata characteristics of RPR --
>> state transitions, context management, absorbability, etc. -- tend to
>> require a much larger volume of test cases than typical RDBMS/SQL
>> features.  This seems unavoidable given the combinatorial nature of
>> NFA execution.

I agree. A complex feature requires complex tests. We cannot avoid it.

BTW, I have tested RPR with large partition/reduced frame (up to 100k
rows). Following query took over 3 minutes on my laptop.

EXPLAIN (ANALYZE)
SELECT aid, count(*) OVER w
FROM generate_series(1,100000) AS g(aid) WHERE aid <= 100000
WINDOW w AS (
ORDER BY aid
ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
AFTER MATCH SKIP PAST LAST ROW
INITIAL
PATTERN (START UP+)
DEFINE START AS TRUE, UP AS aid > PREV(aid)
);
                                                                 QUERY PLAN                                                                 
--------------------------------------------------------------------------------------------------------------------------------------------
 WindowAgg  (cost=4337.40..4504.08 rows=33333 width=14) (actual time=224171.589..224320.055 rows=100000.00 loops=1)
   Window: w AS (ORDER BY aid ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
   Pattern: start up+
   Storage: Disk  Maximum Storage: 4096kB
   NFA States: 200000 peak, 5000050001 total, 0 merged
   NFA Contexts: 100001 peak, 100001 total, 1 pruned
   NFA: 1 matched (len 100000/100000/100000.0), 0 mismatched
   NFA: 0 absorbed, 99998 skipped (len 2/99999/50000.5)
   Buffers: shared hit=3, temp read=400968 written=391
   ->  Sort  (cost=3754.09..3837.42 rows=33333 width=4) (actual time=14.938..38.911 rows=100000.00 loops=1)
         Sort Key: aid
         Sort Method: quicksort  Memory: 3073kB
         Buffers: shared hit=3, temp read=171 written=171
         ->  Function Scan on generate_series g  (cost=0.00..1250.00 rows=33333 width=4) (actual time=5.799..11.672 rows=100000.00 loops=1)
               Filter: (aid <= 100000)
               Buffers: temp read=171 written=171
 Planning:
   Buffers: shared hit=6 read=7
 Planning Time: 0.185 ms
 Execution Time: 224324.532 ms
 
It ran without eating all memory on my box (according to top command,
it was around 200MB). Great! However it took nearly 4 minutes to
complete the query.  I think this is because there are 100k rows
partition/reduced frame, and RPR needs to keep 100k contexts plus 200k
states at the peak. I also ran "perf top" and took following profile.

  19.87%  postgres                               [.] nfa_match
  18.92%  postgres                               [.] nfa_advance_state
  13.60%  postgres                               [.] nfa_cleanup_dead_contexts
  11.25%  postgres                               [.] row_is_in_reduced_frame
  11.06%  postgres                               [.] nfa_state_clone
   5.52%  postgres                               [.] nfa_route_to_elem
   4.39%  postgres                               [.] nfa_add_state_unique.isra.0
   4.32%  postgres                               [.] nfa_state_alloc
   2.36%  libc.so.6                              [.] __memset_avx2_unaligned_erms
   0.63%  postgres                               [.] memset@plt

The reason for nfa_match is called so many times is, it's in a a loop
which iterate over all the contexts:

	for (ctx = winstate->nfaContext; ctx != NULL; ctx = ctx->next)

I don't try to say we should enhance nfa modules in the case when
there is a large reduced frame, because I don't think this is a
realistic use case and I doubt it's worth to fight against unrealistic
scenario.

Note that even if the partition size is large, if we change the PATTERN to:

PATTERN (START UP{,3})

which generates only 4 rows reduced frames, we get very good
performance:

test=# test=# test=# EXPLAIN (ANALYZE)
SELECT aid, count(*) OVER w
FROM generate_series(1,100000) AS g(aid) WHERE aid <= 100000
WINDOW w AS (
ORDER BY aid
ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
AFTER MATCH SKIP PAST LAST ROW
INITIAL
PATTERN (START UP{,3})
DEFINE START AS TRUE, UP AS aid > PREV(aid)
);
---------------------------------------------------------------------------------------------------------------------------------------------
 WindowAgg  (cost=4337.40..4504.08 rows=33333 width=14) (actual time=36.586..91.988 rows=100000.00 loops=1)
   Window: w AS (ORDER BY aid ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
   Pattern: start up{0,3}
   Storage: Memory  Maximum Storage: 17kB
   NFA States: 8 peak, 325001 total, 0 merged
   NFA Contexts: 5 peak, 100001 total, 0 pruned
   NFA: 25000 matched (len 4/4/4.0), 0 mismatched
   NFA: 0 absorbed, 75000 skipped (len 1/3/2.0)
   Buffers: temp read=171 written=171
   ->  Sort  (cost=3754.09..3837.42 rows=33333 width=4) (actual time=36.561..40.933 rows=100000.00 loops=1)
         Sort Key: aid
         Sort Method: quicksort  Memory: 3073kB
         Buffers: temp read=171 written=171
         ->  Function Scan on generate_series g  (cost=0.00..1250.00 rows=33333 width=4) (actual time=14.297..28.692 rows=100000.00 loops=1)
               Filter: (aid <= 100000)
               Buffers: temp read=171 written=171
 Planning Time: 0.087 ms
 Execution Time: 94.810 ms
(18 rows)

I think this is more realistic use case than former. If I were
correct, we don't need to work on current nfa code to squeeze better
performance for unrealistic 100k reduced frame case. I maybe wrong
and I would love to hear from others especially, Henson, Vik, Jacob.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  200. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-09T08:03:23Z

    Hi Tatsuo,

Thank you for the thorough testing and performance profiling!

BTW, I have tested RPR with large partition/reduced frame (up to 100k
> rows). Following query took over 3 minutes on my laptop.
>

The profiling results are very insightful and help us understand where
the time is spent in extreme cases.


> I don't try to say we should enhance nfa modules in the case when
> there is a large reduced frame, because I don't think this is a
> realistic use case and I doubt it's worth to fight against unrealistic
> scenario.
>

I agree with your assessment. Let me clarify:

The pattern itself (START UP+) is realistic - it's a common pattern for
detecting upward trends. However, 100k+ consecutive matches without
interruption is not realistic. Your UP{,3} example (94ms for 100k partition)
demonstrates that the current implementation performs very well for
realistic
use cases.


> I think this is more realistic use case than former. If I were
> correct, we don't need to work on current nfa code to squeeze better
> performance for unrealistic 100k reduced frame case. I maybe wrong
> and I would love to hear from others especially, Henson, Vik, Jacob.


I agree that we should prioritize realistic use cases. That said, there are
potential optimizations that could help:

1. Anchored pattern absorption (see my earlier message:

https://www.postgresql.org/message-id/CAAAe_zAEg7sVM%3DWDwXMyE-odGmQyXSVi5ZzWgye6SupSjdMKpg%40mail.gmail.com
)

2. Alt-pruning: In patterns like "A* | B*", once the higher-priority A
branch
   has a confirmed match, the B branch can be pruned immediately. Even if B
   could continue extending to a longer match, it can never be selected due
to
   lexical order semantics—A will always win. This proactive pruning
respects
   SQL standard semantics while reducing unnecessary state expansion.

However, given the complexity of NFA internals, I believe we should take a
step-by-step approach:

1. First, stabilize the current RPR patch and prepare it for review
2. Then, consider optimizations as separate follow-up patches
3. Each optimization should be well-tested and reviewed independently

This approach reduces risk and makes review more manageable. The fact that
the current implementation handles even unrealistic 100k-row cases without
crashing (just slowly) shows it's already robust.

What do you think about this phased approach?

Best regards,
Henson

P.S. I discovered a crash bug that was introduced in the latest patch
refactoring. The issue occurs with nested alternation patterns like
(A+ | (A | B)+)*, where infinite recursion happens in nfa_advance_alt when
the inner BEGIN(+)'s skip jump is followed as an ALT branch pointer. I will
include the fix in the next patch update.

  201. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-09T10:02:09Z

    Hi Henson,

> Hi Tatsuo,
> 
> Thank you for the thorough testing and performance profiling!
> 
> BTW, I have tested RPR with large partition/reduced frame (up to 100k
>> rows). Following query took over 3 minutes on my laptop.
>>
> 
> The profiling results are very insightful and help us understand where
> the time is spent in extreme cases.

Glad to hear that the data is useful.

> I agree with your assessment. Let me clarify:
> 
> The pattern itself (START UP+) is realistic - it's a common pattern for
> detecting upward trends. However, 100k+ consecutive matches without
> interruption is not realistic. Your UP{,3} example (94ms for 100k partition)
> demonstrates that the current implementation performs very well for
> realistic
> use cases.

Yes, I agree.

> I agree that we should prioritize realistic use cases. That said, there are
> potential optimizations that could help:
> 
> 1. Anchored pattern absorption (see my earlier message:
> 
> https://www.postgresql.org/message-id/CAAAe_zAEg7sVM%3DWDwXMyE-odGmQyXSVi5ZzWgye6SupSjdMKpg%40mail.gmail.com
> )
> 
> 2. Alt-pruning: In patterns like "A* | B*", once the higher-priority A
> branch
>    has a confirmed match, the B branch can be pruned immediately. Even if B
>    could continue extending to a longer match, it can never be selected due
> to
>    lexical order semantics―A will always win. This proactive pruning
> respects
>    SQL standard semantics while reducing unnecessary state expansion.
> 
> However, given the complexity of NFA internals, I believe we should take a
> step-by-step approach:
> 
> 1. First, stabilize the current RPR patch and prepare it for review
> 2. Then, consider optimizations as separate follow-up patches
> 3. Each optimization should be well-tested and reviewed independently
> 
> This approach reduces risk and makes review more manageable. The fact that
> the current implementation handles even unrealistic 100k-row cases without
> crashing (just slowly) shows it's already robust.
> 
> What do you think about this phased approach?

I totally agree.

> Best regards,
> Henson
> 
> P.S. I discovered a crash bug that was introduced in the latest patch
> refactoring. The issue occurs with nested alternation patterns like
> (A+ | (A | B)+)*, where infinite recursion happens in nfa_advance_alt when
> the inner BEGIN(+)'s skip jump is followed as an ALT branch pointer. I will
> include the fix in the next patch update.

Thanks for fixing the issue.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  202. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-11T01:42:56Z

    Hi Ishii-san,

During code review of nodeWindowAgg.c, I discovered that
update_reduced_frame()
can receive the same pos value repeatedly for RANGE/GROUPS frames.


----------------------------------------------------------------------
PATTERN DISCOVERED
----------------------------------------------------------------------

For RANGE/GROUPS frames with peer rows (rows having the same ORDER BY
value):

  Data: id=1,2,3 all have val=10 (peer rows)
        id=4,5 both have val=20 (peer rows)

  RANGE frame processing:
  - id=1 processing: pos=0 (frameheadpos points to first peer)
  - id=2 processing: pos=0 (same frameheadpos)
  - id=3 processing: pos=0 (same frameheadpos)
  - id=4 processing: pos=3
  - id=5 processing: pos=3

ROWS frames always have sequential pos values (0,1,2,3,4...), but
RANGE/GROUPS
can have repeated pos values (0,0,0,3,3...).


----------------------------------------------------------------------
CURRENT IMPLEMENTATION ISSUE
----------------------------------------------------------------------

The current implementation appears to assume sequential processing only,
which
is valid for ROWS frames. Two specific issues found:

1. In update_reduced_frame():
   - The pos <= nfaLastProcessedRow check assumes "already processed pos
doesn't
     need reprocessing", but RANGE/GROUPS require processing the same pos
     multiple times.

2. In nfa_process_row():
   - Frame boundary calculation: ctxFrameEnd = matchStartRow + frameOffset
+ 1
   - This adds frameOffset directly to row position, which works for ROWS
but
     may be incorrect for RANGE (value-based) and GROUPS (group-based)
frames.


----------------------------------------------------------------------
TEST COVERAGE LIMITATION
----------------------------------------------------------------------

The existing RANGE/GROUPS test cases in rpr_base.sql only cover simple
scenarios
that don't reach the frame end, which is why this issue wasn't detected.

Most tests target ROWS frames, with only a small number testing RANGE/GROUPS
frames.


----------------------------------------------------------------------
PROPOSAL
----------------------------------------------------------------------

Rather than directly modifying these functions, we should consider an
alternative approach for non-UNBOUNDED frames:
- Strictly respect the frame head position
- Minimize lower-level optimizations (absorption/pruning)
- Avoid creating subsequent contexts for limited reuse benefit
- For bounded frames, the benefit of context reuse across rows may be
limited
  compared to UNBOUNDED frames where contexts can be reused extensively
- This would simplify RANGE/GROUPS handling at the cost of some optimization

To implement this properly, we need to broaden our understanding of the
execution structure through code review of Window clauses and Aggregate
functions.

Therefore, for this round of nodeWindowAgg.c review:
1. Perform code simplification/refactoring for ROWS frames only
2. Add FIXME comments to RANGE/GROUPS test cases
3. Defer RANGE/GROUPS handling to a future phase

There may be other issues beyond RANGE/GROUPS.
After this code review round is complete, I would like to regroup and
discuss
the direction forward.

Best regards,
Henson

  203. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-11T03:50:32Z

    Hi Henson,

In my understanding RANGE/GROUPS are not allowed when RPR is defined.
(See ISO/IEC 19075-5 section 6.10.2 "ROWS BETWEEN CURRENT ROW AND").
So proper fix would be to error out at the parse/analyze phase if
RANGE/GROUPS are used when RPR is defined.

Vik, Jaconb,
Thoughts?

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

> Hi Ishii-san,
> 
> During code review of nodeWindowAgg.c, I discovered that
> update_reduced_frame()
> can receive the same pos value repeatedly for RANGE/GROUPS frames.
> 
> 
> ----------------------------------------------------------------------
> PATTERN DISCOVERED
> ----------------------------------------------------------------------
> 
> For RANGE/GROUPS frames with peer rows (rows having the same ORDER BY
> value):
> 
>   Data: id=1,2,3 all have val=10 (peer rows)
>         id=4,5 both have val=20 (peer rows)
> 
>   RANGE frame processing:
>   - id=1 processing: pos=0 (frameheadpos points to first peer)
>   - id=2 processing: pos=0 (same frameheadpos)
>   - id=3 processing: pos=0 (same frameheadpos)
>   - id=4 processing: pos=3
>   - id=5 processing: pos=3
> 
> ROWS frames always have sequential pos values (0,1,2,3,4...), but
> RANGE/GROUPS
> can have repeated pos values (0,0,0,3,3...).
> 
> 
> ----------------------------------------------------------------------
> CURRENT IMPLEMENTATION ISSUE
> ----------------------------------------------------------------------
> 
> The current implementation appears to assume sequential processing only,
> which
> is valid for ROWS frames. Two specific issues found:
> 
> 1. In update_reduced_frame():
>    - The pos <= nfaLastProcessedRow check assumes "already processed pos
> doesn't
>      need reprocessing", but RANGE/GROUPS require processing the same pos
>      multiple times.
> 
> 2. In nfa_process_row():
>    - Frame boundary calculation: ctxFrameEnd = matchStartRow + frameOffset
> + 1
>    - This adds frameOffset directly to row position, which works for ROWS
> but
>      may be incorrect for RANGE (value-based) and GROUPS (group-based)
> frames.
> 
> 
> ----------------------------------------------------------------------
> TEST COVERAGE LIMITATION
> ----------------------------------------------------------------------
> 
> The existing RANGE/GROUPS test cases in rpr_base.sql only cover simple
> scenarios
> that don't reach the frame end, which is why this issue wasn't detected.
> 
> Most tests target ROWS frames, with only a small number testing RANGE/GROUPS
> frames.
> 
> 
> ----------------------------------------------------------------------
> PROPOSAL
> ----------------------------------------------------------------------
> 
> Rather than directly modifying these functions, we should consider an
> alternative approach for non-UNBOUNDED frames:
> - Strictly respect the frame head position
> - Minimize lower-level optimizations (absorption/pruning)
> - Avoid creating subsequent contexts for limited reuse benefit
> - For bounded frames, the benefit of context reuse across rows may be
> limited
>   compared to UNBOUNDED frames where contexts can be reused extensively
> - This would simplify RANGE/GROUPS handling at the cost of some optimization
> 
> To implement this properly, we need to broaden our understanding of the
> execution structure through code review of Window clauses and Aggregate
> functions.
> 
> Therefore, for this round of nodeWindowAgg.c review:
> 1. Perform code simplification/refactoring for ROWS frames only
> 2. Add FIXME comments to RANGE/GROUPS test cases
> 3. Defer RANGE/GROUPS handling to a future phase
> 
> There may be other issues beyond RANGE/GROUPS.
> After this code review round is complete, I would like to regroup and
> discuss
> the direction forward.
> 
> Best regards,
> Henson

  204. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-12T01:17:01Z

    Hi Henson,

I found following in rpr_base.sql:

-- {0} is not allowed (min must be >= 1)
SELECT id, val, COUNT(*) OVER w as cnt
FROM rpr_quant
WINDOW w AS (
    ORDER BY id
    ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
    PATTERN (A{0} B)
    DEFINE A AS val > 1000, B AS val > 0
)
ORDER BY id;

However in my uderstanding the SQL standard allows A{0}.

PATTERN (A{0} B)
is equivalant to:
PATTERN (B)

Another interesting PATTERN is:

A{,}

This is equivalant to A{0,}

BTW, after studied this more, I found that A{0,0} is not allowed. In
this form the right hand side number shall be greater than 0. From
ISO/IEC 9075-2 7.9 <row pattern syntax> "Syntax Rules 20)

"If <left brace> <unsigned integer> <comma> <unsigned integer> <right
brace> is specified, then let VUI1 and VUI2 be the values of the first
and second <unsigned integer>'s, respectively. VUI1 shall be less than
or equal to VUI2, and VUI2 shall be greater than 0 (zero)."

However according to the Google, Oracle and Snowflake allows A{0,0}:
they break the standard. So, what do you think PostgreSQL should do
here?  My preference is "always follow the standard". But others might
think differently.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  205. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-12T12:04:52Z

    Hi Tatsuo,

Thank you for the detailed analysis and references to the standard.

However in my uderstanding the SQL standard allows A{0}.
>
> PATTERN (A{0} B)
> is equivalant to:
> PATTERN (B)
>

I agree. We should support A{0} to comply with the standard.


> BTW, after studied this more, I found that A{0,0} is not allowed. In
> this form the right hand side number shall be greater than 0. From
> ISO/IEC 9075-2 7.9 <row pattern syntax> "Syntax Rules 20)
>
> "If <left brace> <unsigned integer> <comma> <unsigned integer> <right
> brace> is specified, then let VUI1 and VUI2 be the values of the first
> and second <unsigned integer>'s, respectively. VUI1 shall be less than
> or equal to VUI2, and VUI2 shall be greater than 0 (zero)."
>


Good catch. I agree we should follow the standard strictly. Since I
don't have direct access to the ISO/IEC 9075-2 document, I trust your
interpretation that A{0,0} should be rejected per SR 20.

However, this raises interesting questions: should we optimize patterns
by removing {0} quantifiers or simplifying them? And if so, how should
we handle patterns that become empty after such optimization?

For example:
- PATTERN (A{0})          → empty pattern
- PATTERN (A{0} B{0})     → empty pattern
- PATTERN (A{0} B)        → PATTERN (B) after optimization

Empty patterns would result in zero-length matches, which our current
implementation explicitly treats as invalid (see initialAdvance flag
logic in nodeWindowAgg.c).

More importantly, I recall that zero-length matches caused serious
issues during development, which is why we added logic to explicitly
avoid them.

The reason I cannot immediately provide a concrete plan for A{0}
support is that I need to deeply understand the semantic meaning of
zero-length matches in the SQL standard first. Without this
understanding, any implementation approach could be fundamentally
flawed.

Specifically, I need to investigate:
- What zero-length matches mean semantically in RPR
- How to handle empty patterns according to the standard
- The correct behavior when a pattern optimizes to nothing

After the current code review phase is complete, I'm also considering
setting up an Oracle test environment to observe how it handles these
edge cases. This could provide valuable insights into the expected
behavior, especially for zero-length matches and empty patterns.

Do you have insights on how the standard handles empty patterns or
zero-length matches?

Best regards,
Henson

  206. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-13T01:14:26Z

    Hi Tatsuo,

However, this raises interesting questions: should we optimize patterns
> by removing {0} quantifiers or simplifying them? And if so, how should
> we handle patterns that become empty after such optimization?
>
> For example:
> - PATTERN (A{0})          → empty pattern
> - PATTERN (A{0} B{0})     → empty pattern
> - PATTERN (A{0} B)        → PATTERN (B) after optimization
>
> Empty patterns would result in zero-length matches, which our current
> implementation explicitly treats as invalid (see initialAdvance flag
> logic in nodeWindowAgg.c).
>
> More importantly, I recall that zero-length matches caused serious
> issues during development, which is why we added logic to explicitly
> avoid them.
>
> The reason I cannot immediately provide a concrete plan for A{0}
> support is that I need to deeply understand the semantic meaning of
> zero-length matches in the SQL standard first. Without this
> understanding, any implementation approach could be fundamentally
> flawed.
>
> Specifically, I need to investigate:
> - What zero-length matches mean semantically in RPR
> - How to handle empty patterns according to the standard
> - The correct behavior when a pattern optimizes to nothing
>
> After the current code review phase is complete, I'm also considering
> setting up an Oracle test environment to observe how it handles these
> edge cases. This could provide valuable insights into the expected
> behavior, especially for zero-length matches and empty patterns.
>

Our current implementation cannot support A{0} due to a structural
limitation.

The reduced_frame_map uses row-based representation (reduced_frame_map[pos]
= val),
which can only express matches consuming at least one row. It cannot
represent
zero-length matches that occur between rows without consuming any row
position.

Patterns like A{0}, A*, or A? can produce zero-length matches with no row
to mark
as RF_FRAME_HEAD and no position to register in the frame map.

We currently prevent this using the initialAdvance flag (nodeWindowAgg.c),
which skips FIN recording during initial epsilon transitions.

Supporting A{0} requires either restructuring reduced_frame_map to handle
virtual positions, or separate handling for zero-length matches. Before
choosing an approach, we need clarity on what the SQL standard expects for
zero-length match semantics (output generation, aggregate behavior, etc.).

Given this structural limitation, I'd like to ask: should we keep the
current
initialAdvance mechanism (which prevents zero-length matches) and handle
A{0}
separately?

Best regards,
Henson

  207. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-13T11:46:50Z

    > Hi Henson,
> 
> In my understanding RANGE/GROUPS are not allowed when RPR is defined.
> (See ISO/IEC 19075-5 section 6.10.2 "ROWS BETWEEN CURRENT ROW AND").
> So proper fix would be to error out at the parse/analyze phase if
> RANGE/GROUPS are used when RPR is defined.

Attached is a patch to disallow RANGE/GROUPS when RPR is used. On top
of your last patches.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  208. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-13T13:01:23Z

    Hi Tatsuo,

Attached is a patch to disallow RANGE/GROUPS when RPR is used. On top
> of your last patches.
>

 Thanks for the patch. I've applied it with a typo fix ("insted" ->
"instead")
and will include it in the next patch series.

Best regards,
Henson

  209. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-14T00:23:10Z

    > Hi Tatsuo,
> 
> However, this raises interesting questions: should we optimize patterns
>> by removing {0} quantifiers or simplifying them? And if so, how should
>> we handle patterns that become empty after such optimization?
>>
>> For example:
>> - PATTERN (A{0})          → empty pattern
>> - PATTERN (A{0} B{0})     → empty pattern
>> - PATTERN (A{0} B)        → PATTERN (B) after optimization

Those optimizations are valid for now because they will produce the
same output tables.  However, if we implement MEASURES, we cannot do
such that optimization. For example,

PATTERN (A{0} B) allows to inquire count(A.*), while PATTERN (B) will
not.

>> Empty patterns would result in zero-length matches, which our current
>> implementation explicitly treats as invalid (see initialAdvance flag
>> logic in nodeWindowAgg.c).
>>
>> More importantly, I recall that zero-length matches caused serious
>> issues during development, which is why we added logic to explicitly
>> avoid them.
>>
>> The reason I cannot immediately provide a concrete plan for A{0}
>> support is that I need to deeply understand the semantic meaning of
>> zero-length matches in the SQL standard first. Without this
>> understanding, any implementation approach could be fundamentally
>> flawed.
>>
>> Specifically, I need to investigate:
>> - What zero-length matches mean semantically in RPR

There are four kinds of pattern matching results in RPR.

1. match:
Pattern successfully matches and maps to a sequence of rows
count aggregate returns > 0
count aggregate as a pattern measure[1] returns > 0
other aggregates return non null
classifier() returns non null result
ordinary column references are null/non null

2. unmatch:
Pattern fails to match a sequence of rows.
count aggregate returns 0
count aggregate as a pattern measure returns 0
other aggregates return null
classifier() returns null
ordinary column references are null/non null

3. no match:
Pattern matching skipped due to AFTER MATCH clause
count aggregate returns 0
count aggregate as a pattern measure returns null
other aggregates return null
classifier() returns null
ordinary column references are null/non null

4. empty match:
Pattern successfully matches but failed to map to a row
count aggregate returns 0
count aggregate as a pattern measure returns 0
other aggregates return null
classifier() returns null
ordinary column references are null/non null

As you can see there's no difference among 2, 3 and 4 in terms of the
output table except "count aggregate as a pattern measure". However we
have not implemented MEASURES, and we can ignore the difference for
now.

[1] e.g. "MEASURES count(*) AS mcount"

>> - How to handle empty patterns according to the standard
>> - The correct behavior when a pattern optimizes to nothing
>>
>> After the current code review phase is complete, I'm also considering
>> setting up an Oracle test environment to observe how it handles these
>> edge cases. This could provide valuable insights into the expected
>> behavior, especially for zero-length matches and empty patterns.
>>
> 
> Our current implementation cannot support A{0} due to a structural
> limitation.
> 
> The reduced_frame_map uses row-based representation (reduced_frame_map[pos]
> = val),
> which can only express matches consuming at least one row. It cannot
> represent
> zero-length matches that occur between rows without consuming any row
> position.

I think we don't need to worry about this ("It cannot represent
zero-length matches that occur between rows without consuming any row
position." problem) In Window RPR (R020) if an empty match is found,
the row position immediately moves to next row. This is because Window
clause requires that number of rows in the input table and the output
table exactly matches. If we insert unmatched rows between the rows,
the number of rows in the output table is different from the number of
rows in the input table, which is not allowed.

> Patterns like A{0}, A*, or A? can produce zero-length matches with no row
> to mark
> as RF_FRAME_HEAD and no position to register in the frame map.

For empty matching rows we just record the starting row position in
reduced_frame_map. We may need to introduce new DEFINE something like
RF_EMPTY though. And row_is_in_reduced_frame returns -1 if RF_EMPTY is
detected. This makes an empty match case behaves like an unmatch case
but it should be ok (see above).

> We currently prevent this using the initialAdvance flag (nodeWindowAgg.c),
> which skips FIN recording during initial epsilon transitions.
> 
> Supporting A{0} requires either restructuring reduced_frame_map to handle
> virtual positions, or separate handling for zero-length matches. Before
> choosing an approach, we need clarity on what the SQL standard expects for
> zero-length match semantics (output generation, aggregate behavior, etc.).

See above. In summary I think we don't need to modify the structure of
reduced_frame_map.

> Given this structural limitation, I'd like to ask: should we keep the
> current
> initialAdvance mechanism (which prevents zero-length matches) and handle
> A{0}
> separately?
> 
> Best regards,
> Henson

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  210. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-14T14:58:10Z

    Hi Tatsuo,

This round focused on reviewing nodeWindowAgg.c -- the NFA executor
-- which was the last piece remaining on my review list.  I've also
added a comprehensive NFA runtime test suite.

The attached patch continues the incremental series on top of v42.
Here are the changes:


FIXME issues documented:

These require non-trivial structural changes, so I've documented
them as FIXME comments for now rather than attempting a fix.

- altPriority tracks only the last ALT choice, so repeated or nested
  ALTs like (A|B)+ cannot correctly implement SQL standard lexical
  ordering.  A full-path classifier structure is needed.

- Cycle prevention condition (count == 0 && min == 0) is insufficient
  for patterns like (A*)* where cycles occur at count > 0.  Currently
  relies on implicit duplicate detection in nfa_add_state_unique.


Bugs fixed:

- Fix nfa_advance_begin() routing order: enter group first (lexically
  first), then skip path (lexically second).

- Add ALT scope boundary check in nfa_advance_alt() and
  computeAbsorbabilityRecursive() to stop branch traversal when
  element depth exits ALT scope.

- Disallow RANGE and GROUPS frame types with row pattern
  recognition, as the SQL standard only requires ROWS.
  Also remove the now-dead frame-type determination code.


NFA executor refactoring (nodeWindowAgg.c):

- Simplify nfa_process_row() phase 1: remove nfa_advance() call
  after forced mismatch at frame boundary, since all states are
  already at VAR positions and mismatch removes them.  Replace
  the redundant phase 3 frame boundary check with Assert.

- Simplify update_reduced_frame() result registration: mismatch
  path now returns early, and the post-processing SKIP mode cleanup
  block (nfa_remove_contexts_up_to vs nfa_context_free) is removed
  since eager pruning handles SKIP PAST LAST ROW and both paths
  now simply call nfa_context_free().

- Replace nfa_remove_contexts_up_to() with eager pruning inside
  nfa_add_matched_state().  When matchEndRow is extended during
  greedy matching, pending contexts within the match range are
  pruned incrementally instead of after match completion.  For
  SKIP PAST LAST ROW patterns like START UP+, this reduces the
  number of live contexts at each row from O(n) to O(1), avoiding
  O(n^2) per-row processing of contexts that would be skipped
  anyway.

- Optimize nfa_states_equal() to compare counts only up to the
  current element's depth instead of the full maxDepth.

- Rename nfa_state_clone -> nfa_state_create,
  nfa_find_context_for_pos -> nfa_get_head_context for clarity.

- Add nfa_record_context_success/skipped/absorbed statistics helpers.

- Remove unused RPRPattern parameter from
  nfa_update_absorption_flags().


Absorbability refactoring (rpr.c):

- Remove parentDepth parameter from isUnboundedStart() and
  computeAbsorbabilityRecursive().  Scope boundaries are now
  determined by element depth comparison instead of an explicit
  parent depth parameter.

- Simplify isUnboundedStart(): check simple VAR case first, then
  GROUP case with a depth-bounded loop instead of a FIN-terminated
  traversal with multiple break conditions.


Test changes:

- Add rpr_nfa.sql: comprehensive NFA runtime test suite covering
  quantifier boundaries (min/max/exact), alternation priority, nested
  patterns, frame boundary variations, INITIAL mode (syntax error
  expected), pathological patterns, context absorption, and FIXME
  reproduction cases.

- Add nth_value out-of-frame tests, ReScan/LATERAL test, and
  nth_value IGNORE NULLS test to rpr.sql.

- Change selected EXPLAIN test queries in rpr_explain.sql from
  EXPLAIN (COSTS OFF) to EXPLAIN (ANALYZE, ...) to verify actual
  NFA execution statistics.

- Fix stale comments across rpr.sql, rpr_base.sql, and rpr_nfa.sql:
  remove resolved BUG annotations, update error messages to match
  actual output, correct optimization result descriptions, and
  standardize Expected comment placement to after SQL statements.


Other changes:

- Run pgindent on RPR source files.  Add /*----------*/ comment
  guards to protect structured comments from reformatting.


Coverage:

I ran gcov on the modified lines (diff-only coverage).  The attached
coverage.zip contains an HTML report.  Summary:

- 18 files, 124 functions, 2238 modified lines analyzed
- Overall: 92.1% line coverage (2061/2238)
- Core files (nodeWindowAgg.c, rpr.c, explain.c, parse_rpr.c):
  98.0-98.5% each
- Node serialization (outfuncs.c, readfuncs.c, equalfuncs.c):
  0% -- these implement RPRPattern serialization for plan caching,
  but regression tests don't exercise prepared statements with RPR
- ruleutils.c: 96.3% -- untested lines are the reluctant quantifier
  display path ({1}?), which is currently rejected at parse time

The node serialization functions (141 lines, 0% coverage) are the
largest untested area.  I'm not sure how to trigger these paths
in the regression test framework.  Any suggestions?

I'll send a separate email within a few days listing the FIXME
issues and other unresolved items from the mailing list discussion
for your review.


Best regards,
Henson

  211. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-15T09:06:52Z

    Hi Henson,

Thanks for the patches! I applied them on top of v42 and rebased
against current master.  Attached are the v43 patches.

> Hi Tatsuo,
> 
> This round focused on reviewing nodeWindowAgg.c -- the NFA executor
> -- which was the last piece remaining on my review list.  I've also
> added a comprehensive NFA runtime test suite.
> 
> The attached patch continues the incremental series on top of v42.
> Here are the changes:
> 
> 
> FIXME issues documented:
> 
> These require non-trivial structural changes, so I've documented
> them as FIXME comments for now rather than attempting a fix.
> 
> - altPriority tracks only the last ALT choice, so repeated or nested
>   ALTs like (A|B)+ cannot correctly implement SQL standard lexical
>   ordering.  A full-path classifier structure is needed.
> 
> - Cycle prevention condition (count == 0 && min == 0) is insufficient
>   for patterns like (A*)* where cycles occur at count > 0.  Currently
>   relies on implicit duplicate detection in nfa_add_state_unique.
> 
> 
> Bugs fixed:
> 
> - Fix nfa_advance_begin() routing order: enter group first (lexically
>   first), then skip path (lexically second).
> 
> - Add ALT scope boundary check in nfa_advance_alt() and
>   computeAbsorbabilityRecursive() to stop branch traversal when
>   element depth exits ALT scope.
> 
> - Disallow RANGE and GROUPS frame types with row pattern
>   recognition, as the SQL standard only requires ROWS.
>   Also remove the now-dead frame-type determination code.
> 
> 
> NFA executor refactoring (nodeWindowAgg.c):
> 
> - Simplify nfa_process_row() phase 1: remove nfa_advance() call
>   after forced mismatch at frame boundary, since all states are
>   already at VAR positions and mismatch removes them.  Replace
>   the redundant phase 3 frame boundary check with Assert.
> 
> - Simplify update_reduced_frame() result registration: mismatch
>   path now returns early, and the post-processing SKIP mode cleanup
>   block (nfa_remove_contexts_up_to vs nfa_context_free) is removed
>   since eager pruning handles SKIP PAST LAST ROW and both paths
>   now simply call nfa_context_free().
> 
> - Replace nfa_remove_contexts_up_to() with eager pruning inside
>   nfa_add_matched_state().  When matchEndRow is extended during
>   greedy matching, pending contexts within the match range are
>   pruned incrementally instead of after match completion.  For
>   SKIP PAST LAST ROW patterns like START UP+, this reduces the
>   number of live contexts at each row from O(n) to O(1), avoiding
>   O(n^2) per-row processing of contexts that would be skipped
>   anyway.
> 
> - Optimize nfa_states_equal() to compare counts only up to the
>   current element's depth instead of the full maxDepth.
> 
> - Rename nfa_state_clone -> nfa_state_create,
>   nfa_find_context_for_pos -> nfa_get_head_context for clarity.
> 
> - Add nfa_record_context_success/skipped/absorbed statistics helpers.
> 
> - Remove unused RPRPattern parameter from
>   nfa_update_absorption_flags().
> 
> 
> Absorbability refactoring (rpr.c):
> 
> - Remove parentDepth parameter from isUnboundedStart() and
>   computeAbsorbabilityRecursive().  Scope boundaries are now
>   determined by element depth comparison instead of an explicit
>   parent depth parameter.
> 
> - Simplify isUnboundedStart(): check simple VAR case first, then
>   GROUP case with a depth-bounded loop instead of a FIN-terminated
>   traversal with multiple break conditions.
> 
> 
> Test changes:
> 
> - Add rpr_nfa.sql: comprehensive NFA runtime test suite covering
>   quantifier boundaries (min/max/exact), alternation priority, nested
>   patterns, frame boundary variations, INITIAL mode (syntax error
>   expected), pathological patterns, context absorption, and FIXME
>   reproduction cases.
> 
> - Add nth_value out-of-frame tests, ReScan/LATERAL test, and
>   nth_value IGNORE NULLS test to rpr.sql.
> 
> - Change selected EXPLAIN test queries in rpr_explain.sql from
>   EXPLAIN (COSTS OFF) to EXPLAIN (ANALYZE, ...) to verify actual
>   NFA execution statistics.
> 
> - Fix stale comments across rpr.sql, rpr_base.sql, and rpr_nfa.sql:
>   remove resolved BUG annotations, update error messages to match
>   actual output, correct optimization result descriptions, and
>   standardize Expected comment placement to after SQL statements.
> 
> 
> Other changes:
> 
> - Run pgindent on RPR source files.  Add /*----------*/ comment
>   guards to protect structured comments from reformatting.
>
> Coverage:
> 
> I ran gcov on the modified lines (diff-only coverage).  The attached
> coverage.zip contains an HTML report.  Summary:
> 
> - 18 files, 124 functions, 2238 modified lines analyzed
> - Overall: 92.1% line coverage (2061/2238)
> - Core files (nodeWindowAgg.c, rpr.c, explain.c, parse_rpr.c):
>   98.0-98.5% each
> - Node serialization (outfuncs.c, readfuncs.c, equalfuncs.c):
>   0% -- these implement RPRPattern serialization for plan caching,
>   but regression tests don't exercise prepared statements with RPR
> - ruleutils.c: 96.3% -- untested lines are the reluctant quantifier
>   display path ({1}?), which is currently rejected at parse time

Thanks for the report.

> The node serialization functions (141 lines, 0% coverage) are the
> largest untested area.  I'm not sure how to trigger these paths
> in the regression test framework.  Any suggestions?

I think we can leave it as it is, until reluctant quantifier is
implemented.

> I'll send a separate email within a few days listing the FIXME
> issues and other unresolved items from the mailing list discussion
> for your review.

Looking forward to reading your email.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  212. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-16T15:09:04Z

    Hi Tatsuo,

Thanks for rebasing to v43!

I noticed the Cirrus CI 32-bit build (test_world_32) failed on three
tests, all due to the same root cause in rpr_explain.out:

  Sort Method: quicksort  Memory: 27kB  (expected, 64-bit)
  Sort Method: quicksort  Memory: 22kB  (actual, 32-bit)

The rpr_explain_filter() function was already normalizing the RPR
Storage memory values, but missed the Sort node's "Memory: NNkB"
which also varies between 32-bit and 64-bit due to pointer/struct
size differences.

The attached incremental patch adds Sort Method memory normalization
to rpr_explain_filter(), using the same approach as the existing
Storage and Maximum Storage filters.

> The node serialization functions (141 lines, 0% coverage) are the
> > largest untested area.  I'm not sure how to trigger these paths
> > in the regression test framework.  Any suggestions?
>
> I think we can leave it as it is, until reluctant quantifier is
> implemented.
>

Agreed.


> > I'll send a separate email within a few days listing the FIXME
> > issues and other unresolved items from the mailing list discussion
> > for your review.
>
> Looking forward to reading your email.


Best regards,
Henson

  213. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-16T23:20:20Z

    Hi Henson,

> Hi Tatsuo,
> 
> Thanks for rebasing to v43!
> 
> I noticed the Cirrus CI 32-bit build (test_world_32) failed on three
> tests, all due to the same root cause in rpr_explain.out:
> 
>   Sort Method: quicksort  Memory: 27kB  (expected, 64-bit)
>   Sort Method: quicksort  Memory: 22kB  (actual, 32-bit)

I noticed that too.

> The rpr_explain_filter() function was already normalizing the RPR
> Storage memory values, but missed the Sort node's "Memory: NNkB"
> which also varies between 32-bit and 64-bit due to pointer/struct
> size differences.
> 
> The attached incremental patch adds Sort Method memory normalization
> to rpr_explain_filter(), using the same approach as the existing
> Storage and Maximum Storage filters.

Looks good fix to me. Is that Ok for you to not release v44 patch
immediately?  Our patch set is very large and I hesitate to post a new
patch set too offten.

>> The node serialization functions (141 lines, 0% coverage) are the
>> > largest untested area.  I'm not sure how to trigger these paths
>> > in the regression test framework.  Any suggestions?
>>
>> I think we can leave it as it is, until reluctant quantifier is
>> implemented.
>>
> 
> Agreed.

Thanks for agreeing on this.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  214. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-16T23:27:42Z

    Hi Tatsuo,

As promised, here is a summary of the open items and my proposed
plan for the remaining work.


1. FIXME bugs (to fix within the patch)

These are correctness issues that should be resolved before commit.

[FIXME] Cycle prevention insufficient

  The cycle prevention condition (count == 0 && min == 0) is
  insufficient for patterns like (A*)* where cycles occur at
  count > 0.  Currently relies on implicit duplicate detection
  in nfa_add_state_unique.  An explicit cycle detection logic
  needs to be added.

[FIXME] altPriority -- incomplete lexical ordering

  altPriority tracks only the last ALT choice, so repeated or
  nested ALTs like (A|B)+ cannot correctly implement SQL standard
  lexical ordering.  We may need to maintain an internal classifier
  that tracks the entire ALT path to correctly determine lexical
  priority across repeated alternations.  I'm considering an
  elemIdx-based classifier that records the sequence of element
  positions through ALT branches -- from elemIdx we can derive
  altPriority, varId, and variable name, so it carries more
  information than a varId-based approach, which cannot distinguish
  branches when the same variable appears in multiple ALT
  alternatives.  Since the classifier sequence grows with each
  ALT choice, state forks will need an efficient sharing structure
  (e.g., chunk tree with refcounting) to avoid memory explosion.
  The details still need refinement and verification, so it may
  take some time.


2. Test and documentation (within the patch)

Node serialization 0% coverage

  outfuncs.c/readfuncs.c/equalfuncs.c.  You suggested leaving
  this until reluctant quantifiers are implemented.  This depends
  on the scope decision in section 4 -- if reluctant quantifiers
  are included, we should address coverage then.

Test reorganization

  - Remove tests from rpr.sql that duplicate coverage in rpr_base.sql
    and rpr_nfa.sql
  - Restructure tests around SQL-centric scenarios rather than
    internal implementation details
  - Add real-world scenario tests (e.g., stock V-shape, sessionization)

Documentation review

  - Review and enhance the existing SGML documentation
  - Ensure all implemented features are accurately documented


3. Performance optimizations (separate follow-up patches)

These require thorough analysis and research, but since they are
pure performance improvements with no compatibility impact, they
can be submitted as independent patches afterward.

State pruning optimization

  Active NFA states with lower lexical priority than an already
  confirmed match candidate could be pruned early.  Correctness
  is hard to guarantee for all pattern combinations.  Documented
  as a future research item.

Anchored pattern absorption optimization

  PREFIX elements (e.g., START in "START A+ B") block absorption,
  causing O(n^2) regression for anchored patterns.  A draft design
  exists: track an alternate "shadow" path that skips PREFIX and
  starts at the BODY region, enabling absorption eligibility checks
  while the original path processes PREFIX normally.  This keeps
  concurrent contexts bounded to PREFIX_length + 1, maintaining
  O(n) complexity.  The design needs further refinement before
  implementation.


4. Unimplemented features

The following R020 features are listed in the commit message as not
yet implemented.  Could you let me know which ones you'd like to
include in this patch versus deferring to follow-up patches?

  Requires per-match state tracking infrastructure:
  - MEASURE -- new clause, pattern variable references in target list,
    and per-match aggregate evaluation within the NFA executor
  - CLASSIFIER -- requires per-row variable binding in match state;
    the altPriority fix in section 1 would provide part of this
    infrastructure

  Quantifier/pattern extensions:
  - Reluctant quantifiers (*? etc.) -- parsing exists, runtime error
  - {- -} (exclusion)
  - Empty pattern -- A{0}, () empty pattern.  Should we implement
    A{0} in this patch and defer empty match semantics (unmatch vs
    empty match distinction) to when MEASURE is implemented?
    A{0,0}: standard forbids, Oracle/Snowflake allow.
    Should we follow the standard and reject A{0,0}?
  - PERMUTE -- can be expanded to alternations at parse time

  Navigation/control:
  - SEEK
  - AFTER MATCH SKIP TO / FIRST / LAST
    (only SKIP PAST LAST ROW and SKIP TO NEXT ROW are supported)

  Row pattern functions:
  - SUBSET -- variable grouping for aggregate evaluation
  - FIRST, LAST

  Out of scope:
  - MATCH_RECOGNIZE (R010)


I'd like your thoughts on the following scope questions:
  - Reluctant quantifiers: in scope for this patch?
  - A{0}: proceed with implementation?
  - () empty pattern: also in scope, or defer?
  - A{0,0}: reject per standard, or allow like Oracle/Snowflake?

There are quite a few items listed above, and I'd like to
make sure I'm working on the right things in the right order.
If you could give me some guidance on which items to prioritize
and how you'd like to see these addressed, that would be very
helpful.

Best regards,
Henson

  215. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-16T23:30:26Z

    Hi Tatsuo,

> Looks good fix to me. Is that Ok for you to not release v44 patch
> immediately?  Our patch set is very large and I hesitate to post a new
> patch set too offten.

Sure, no problem.  I think it's better to bundle it with the next
round of changes rather than posting a separate v44 just for this fix.
There's no real urgency since the 32-bit issue only affects the
explain test output and doesn't impact functionality.

Best regards,
Henson

  216. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-17T06:39:22Z

    Hi Henson,

> Anchored pattern absorption optimization
> 
>   PREFIX elements (e.g., START in "START A+ B") block absorption,
>   causing O(n^2) regression for anchored patterns.  A draft design
>   exists: track an alternate "shadow" path that skips PREFIX and
>   starts at the BODY region, enabling absorption eligibility checks
>   while the original path processes PREFIX normally.  This keeps
>   concurrent contexts bounded to PREFIX_length + 1, maintaining
>   O(n) complexity.  The design needs further refinement before
>   implementation.

What do you mean by "Anchored pattern" here? I am asking because R010
(RPR in Window clause) does not allow to use anchors (^ and $) in
PATTERN clause.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  217. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-17T06:44:57Z

    Hi Tatsuo,


> What do you mean by "Anchored pattern" here? I am asking because R010
> (RPR in Window clause) does not allow to use anchors (^ and $) in
> PATTERN clause.
>

Good catch - the term "anchored pattern" is indeed confusing here.
You're absolutely right that SQL:2023 R010 does not allow regex anchors
(^ and $) in the PATTERN clause.
In this context, I was using "anchored" to mean "a pattern that starts
with a PREFIX element" - not referring to regex anchors at all. The
example "START A+ B" has a PREFIX element (START) that "anchors" the
pattern matching to begin at a specific point.
To avoid confusion with SQL regex anchors, I suggest we change the
terminology:

"Anchored pattern" → "Pattern with PREFIX" or "PREFIX-led pattern"
"Anchored pattern absorption" → "PREFIX pattern absorption optimization"

The optimization issue remains the same: PREFIX elements block the
absorption mechanism, requiring the "shadow path" approach to maintain
O(n) complexity while processing patterns like "START A+ B".
Does this clarification make sense?

Best regards,
Henson

  218. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-17T09:00:53Z

    Hi Henson,

>> What do you mean by "Anchored pattern" here? I am asking because R010
>> (RPR in Window clause) does not allow to use anchors (^ and $) in
>> PATTERN clause.
>>
> 
> Good catch - the term "anchored pattern" is indeed confusing here.
> You're absolutely right that SQL:2023 R010 does not allow regex anchors
> (^ and $) in the PATTERN clause.
> In this context, I was using "anchored" to mean "a pattern that starts
> with a PREFIX element" - not referring to regex anchors at all. The
> example "START A+ B" has a PREFIX element (START) that "anchors" the
> pattern matching to begin at a specific point.

Ok.

> To avoid confusion with SQL regex anchors, I suggest we change the
> terminology:
> 
> "Anchored pattern" → "Pattern with PREFIX" or "PREFIX-led pattern"

I like "Pattern with PREFIX".

> "Anchored pattern absorption" → "PREFIX pattern absorption optimization"

Looks good.

> The optimization issue remains the same: PREFIX elements block the
> absorption mechanism, requiring the "shadow path" approach to maintain
> O(n) complexity while processing patterns like "START A+ B".
> Does this clarification make sense?

Yes!

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  219. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-17T09:34:28Z

    Hi Henson,

> As promised, here is a summary of the open items and my proposed
> plan for the remaining work.

> 1. FIXME bugs (to fix within the patch)
> 2. Test and documentation (within the patch)
> 3. Performance optimizations (separate follow-up patches)

1-3 look good.

> 4. Unimplemented features
> 
> The following R020 features are listed in the commit message as not
> yet implemented.  Could you let me know which ones you'd like to
> include in this patch versus deferring to follow-up patches?
> 
>   Requires per-match state tracking infrastructure:
>   - MEASURE -- new clause, pattern variable references in target list,
>     and per-match aggregate evaluation within the NFA executor
>   - CLASSIFIER -- requires per-row variable binding in match state;
>     the altPriority fix in section 1 would provide part of this
>     infrastructure

I thought we can defere these after the first committed version.  I
think these are too large to add the current patch sets.

>   Quantifier/pattern extensions:
>   - Reluctant quantifiers (*? etc.) -- parsing exists, runtime error

Reluctant quantifiers seem to have important usecases (i.e. the
shortest path). But if implementation is too difficult, we should
defer to after the first version.

>   - {- -} (exclusion)
>   - Empty pattern -- A{0}, () empty pattern.  Should we implement
>     A{0} in this patch and defer empty match semantics (unmatch vs
>     empty match distinction) to when MEASURE is implemented?

Yes, I think so. We should defer.

>     A{0,0}: standard forbids, Oracle/Snowflake allow.
>     Should we follow the standard and reject A{0,0}?

I think we should follow the standard.

>   - PERMUTE -- can be expanded to alternations at parse time

I though we could defer after the first commit. Do you wish to
implemenet it now?

>   Navigation/control:
>   - SEEK
>   - AFTER MATCH SKIP TO / FIRST / LAST
>     (only SKIP PAST LAST ROW and SKIP TO NEXT ROW are supported)

Let's defer them.

>   Row pattern functions:
>   - SUBSET -- variable grouping for aggregate evaluation
>   - FIRST, LAST

Let's defer them.

>   Out of scope:
>   - MATCH_RECOGNIZE (R010)
> 
> 
> I'd like your thoughts on the following scope questions:
>   - Reluctant quantifiers: in scope for this patch?

I found reluctant quantifiers are useful but also I don't want to make
patch sets far bigger. How do yo estimate the difficulty and the size
of the code for reluctant quantifiers?

>   - A{0}: proceed with implementation?
>   - () empty pattern: also in scope, or defer?

I think we can defer.

>   - A{0,0}: reject per standard, or allow like Oracle/Snowflake?

Let's reject per standard.

> There are quite a few items listed above, and I'd like to
> make sure I'm working on the right things in the right order.
> If you could give me some guidance on which items to prioritize
> and how you'd like to see these addressed, that would be very
> helpful.

Here are my thoughts. What do you think?

BTW, while researching the standard, I am wondering whether empty
DEFINE clause is allowed (DEFINE without any variable definition at
all). According to ISO/IEC 9075-2:

<row pattern definition list> ::=
 <row pattern definition> [ { <comma> <row pattern definition> }...]

<row pattern definition> ::=
 <row pattern definition variable name> AS < row pattern definition search condition>
:
:

So, it seems a bare DEFINE clause is not allowed. However searching
goole, it says that the standard allow a bare DEFINE. Maybe it's a
hallucination?
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  220. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-18T10:56:53Z

    From: Henson Choi <assam258(at)gmail(dot)com>
To: Tatsuo Ishii <ishii(at)postgresql(dot)org>
Cc: vik(at)postgresfriends(dot)org, er(at)xs4all(dot)nl,
jacob(dot)champion(at)enterprisedb(dot)com,
david(dot)g(dot)johnston(at)gmail(dot)com, peter(at)eisentraut(dot)org,
pgsql-hackers(at)postgresql(dot)org
Subject: Re: Row pattern recognition

Hi Tatsuo,

Thanks for the feedback!  Here are my responses to your questions.


> Reluctant quantifiers seem to have important usecases (i.e. the
> shortest path). But if implementation is too difficult, we should
> defer to after the first version.
>
> How do you estimate the difficulty and the size of the code for
> reluctant quantifiers?

The core difficulty is not in nfa_advance() itself but in matched
state replacement: once a greedy match is confirmed, replacing it
with a shorter reluctant match requires updating matchedState, which
the current structure does not support cleanly.

This is the same design gap that underlies the altPriority FIXME:
both problems require rethinking how the confirmed match state
interacts with the active NFA state list.  I think it is worth
solving them together.

I would prefer to work on the altPriority redesign first and include
reluctant quantifiers and state pruning as part of that work if the
design allows it cleanly.

If I encounter design issues that are difficult to resolve, I will
come back to discuss before proceeding.


> >   - {- -} (exclusion)
> >   - Empty pattern -- A{0}, () empty pattern.  Should we implement
> >     A{0} in this patch and defer empty match semantics (unmatch vs
> >     empty match distinction) to when MEASURE is implemented?
>
> Yes, I think so. We should defer.
>
> >   - A{0}: proceed with implementation?
> >   - () empty pattern: also in scope, or defer?
>
> I think we can defer.

Just to confirm: my understanding is that A{0} is in scope for this
patch, while () empty pattern and empty match semantics are deferred.
Is that correct?  If so, I plan to implement A{0} together with the
cycle detection fix for patterns like (A*)*, since both involve
zero-length iteration handling.


> I though we could defer after the first commit. Do you wish to
> implement it now?
> [regarding PERMUTE]

I am happy to defer PERMUTE.


> 1-3 look good.

I will address test reorganization and documentation review after the bug
fixes and feature implementations are complete.


> BTW, while researching the standard, I am wondering whether empty
> DEFINE clause is allowed (DEFINE without any variable definition at
> all).  [...] So, it seems a bare DEFINE clause is not allowed.
> However searching google, it says that the standard allow a bare
> DEFINE. Maybe it's a hallucination?

That seems right.  DEFINE with no definitions is not allowed; the
parser rejects it syntactically.

On a related note: since the standard treats variables not listed in
DEFINE as implicitly TRUE, should we allow PATTERN without DEFINE?


Best regards,
Henson

  221. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-18T22:28:31Z

    >> Reluctant quantifiers seem to have important usecases (i.e. the
>> shortest path). But if implementation is too difficult, we should
>> defer to after the first version.
>>
>> How do you estimate the difficulty and the size of the code for
>> reluctant quantifiers?
> 
> The core difficulty is not in nfa_advance() itself but in matched
> state replacement: once a greedy match is confirmed, replacing it
> with a shorter reluctant match requires updating matchedState, which
> the current structure does not support cleanly.
> 
> This is the same design gap that underlies the altPriority FIXME:
> both problems require rethinking how the confirmed match state
> interacts with the active NFA state list.  I think it is worth
> solving them together.

Agreed.

> I would prefer to work on the altPriority redesign first and include
> reluctant quantifiers and state pruning as part of that work if the
> design allows it cleanly.
> 
> If I encounter design issues that are difficult to resolve, I will
> come back to discuss before proceeding.

Sounds like a perfect plan!

>> >   - {- -} (exclusion)
>> >   - Empty pattern -- A{0}, () empty pattern.  Should we implement
>> >     A{0} in this patch and defer empty match semantics (unmatch vs
>> >     empty match distinction) to when MEASURE is implemented?
>>
>> Yes, I think so. We should defer.
>>
>> >   - A{0}: proceed with implementation?
>> >   - () empty pattern: also in scope, or defer?
>>
>> I think we can defer.
> 
> Just to confirm: my understanding is that A{0} is in scope for this
> patch, while () empty pattern and empty match semantics are deferred.
> Is that correct?  If so, I plan to implement A{0} together with the
> cycle detection fix for patterns like (A*)*, since both involve
> zero-length iteration handling.

Thanks for clarification. That's my understanding too.

>> I though we could defer after the first commit. Do you wish to
>> implement it now?
>> [regarding PERMUTE]
> 
> I am happy to defer PERMUTE.

Ok.

>> 1-3 look good.
> 
> I will address test reorganization and documentation review after the bug
> fixes and feature implementations are complete.

I can work on the documentation review part.

>> BTW, while researching the standard, I am wondering whether empty
>> DEFINE clause is allowed (DEFINE without any variable definition at
>> all).  [...] So, it seems a bare DEFINE clause is not allowed.
>> However searching google, it says that the standard allow a bare
>> DEFINE. Maybe it's a hallucination?
> 
> That seems right.  DEFINE with no definitions is not allowed; the
> parser rejects it syntactically.
> 
> On a related note: since the standard treats variables not listed in
> DEFINE as implicitly TRUE, should we allow PATTERN without DEFINE?

ISO/IEC 19075-5 stats that DEFINE is not optional (Table 18 - Row
pattern recognition in windows - syntax summary) So we should not
allow PATTERN without DEFINE.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  222. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-19T04:47:51Z

    Hi Henson,

> Thanks for the patches! I applied them on top of v42 and rebased
> against current master.  Attached are the v43 patches.

I noticed that current RPR patches allows to use PREV/NEXT anywhere in
an SQL statement where normal functions are allowed. According to the
standard, PREV/NEXT are only allowed in a DEFINE clause. Attached is a
patch to follow the standard regarding PREV/NEXT.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  223. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-19T05:00:21Z

    Hi Tatsuo,

I noticed that current RPR patches allows to use PREV/NEXT anywhere in
> an SQL statement where normal functions are allowed. According to the
> standard, PREV/NEXT are only allowed in a DEFINE clause. Attached is a
> patch to follow the standard regarding PREV/NEXT.
>

Thanks for the patch! I applied it to my local branch and all
regression tests pass.

Best regards,
Henson

  224. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-22T11:09:52Z

    Hi Tatsuo,

Here are two incremental patches on top of v43 + our previous two.


nocfbot-0003: Fix ALT lexical ordering via DFS early termination

The altPriority FIXME turned out to have a simple solution.  The key
insight is that the NFA state list is already in lexical order from
DFS traversal, so when a state reaches FIN, all remaining states in
the list have worse lexical priority and can be pruned immediately.

This makes the altPriority field unnecessary -- DFS traversal order
itself guarantees correct lexical ordering.  The patch removes
altPriority from RPRNFAState entirely and adds early termination
in nfa_advance(): when a new FIN is reached, the remaining states
are freed and processing stops.

This also implements the state pruning optimization I mentioned
earlier -- it falls out naturally from the same mechanism.

Changes:
- Remove altPriority field from RPRNFAState and all call sites
- Add early termination in nfa_advance() on new FIN arrival
- Simplify nfa_add_matched_state() to unconditional replacement
- Fix outer END count increment for quantified VARs in
  nfa_advance_var()
- Remove FIXME 1 test cases, add test_alt_lexical_order test
- Add EXPLAIN ANALYZE test verifying early termination statistics


nocfbot-0004: Implement reluctant quantifiers

With the DFS early termination infrastructure from nocfbot-0003,
reluctant quantifiers become straightforward: reverse the DFS
traversal order so that shorter matches are explored first, then
apply the same early termination to stop at the shortest match.

Greedy explores enter/loop before skip/exit; reluctant reverses
this to skip/exit before enter/loop.  When the preferred (shorter)
path reaches FIN, the longer path is pruned immediately.

Changes:
- Remove parser error rejecting reluctant quantifier syntax
- Extend tryUnwrapGroup() to propagate reluctant flag when
  unwrapping single-child groups: (A)+? -> A+?
- Add reluctant branching in nfa_advance_begin(),
  nfa_advance_end(), and nfa_advance_var() with per-branch
  early termination
- Add tests in rpr_base.sql, rpr_nfa.sql, and rpr.sql covering
  basic reluctant semantics, quantifier boundaries, interaction
  with greedy quantifiers, and nested/alternation combinations


> I found reluctant quantifiers are useful but also I don't want to make
> patch sets far bigger. How do you estimate the difficulty and the size
> of the code for reluctant quantifiers?

You asked earlier how I estimated the difficulty of reluctant
quantifiers.  It turned out the answer was subtraction, not
addition -- removing altPriority and simplifying the match logic
first made reluctant quantifiers almost trivial to add on top.

Next I plan to work on the remaining FIXME (cycle prevention for
patterns like (A*)*), A{0} implementation, and test reorganization.


Best regards,
Henson

>

  225. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-23T10:26:46Z

    Hi Henson,

> Hi Tatsuo,
> 
> Here are two incremental patches on top of v43 + our previous two.

I have tried v43 + those patches and it passed the regression test.

> nocfbot-0003: Fix ALT lexical ordering via DFS early termination
> 
> The altPriority FIXME turned out to have a simple solution.  The key
> insight is that the NFA state list is already in lexical order from
> DFS traversal, so when a state reaches FIN, all remaining states in
> the list have worse lexical priority and can be pruned immediately.
> 
> This makes the altPriority field unnecessary -- DFS traversal order
> itself guarantees correct lexical ordering.  The patch removes
> altPriority from RPRNFAState entirely and adds early termination
> in nfa_advance(): when a new FIN is reached, the remaining states
> are freed and processing stops.
> 
> This also implements the state pruning optimization I mentioned
> earlier -- it falls out naturally from the same mechanism.
> 
> Changes:
> - Remove altPriority field from RPRNFAState and all call sites
> - Add early termination in nfa_advance() on new FIN arrival
> - Simplify nfa_add_matched_state() to unconditional replacement
> - Fix outer END count increment for quantified VARs in
>   nfa_advance_var()
> - Remove FIXME 1 test cases, add test_alt_lexical_order test
> - Add EXPLAIN ANALYZE test verifying early termination statistics

Sounds good.

> nocfbot-0004: Implement reluctant quantifiers
> 
> With the DFS early termination infrastructure from nocfbot-0003,
> reluctant quantifiers become straightforward: reverse the DFS
> traversal order so that shorter matches are explored first, then
> apply the same early termination to stop at the shortest match.
> 
> Greedy explores enter/loop before skip/exit; reluctant reverses
> this to skip/exit before enter/loop.  When the preferred (shorter)
> path reaches FIN, the longer path is pruned immediately.
> 
> Changes:
> - Remove parser error rejecting reluctant quantifier syntax
> - Extend tryUnwrapGroup() to propagate reluctant flag when
>   unwrapping single-child groups: (A)+? -> A+?
> - Add reluctant branching in nfa_advance_begin(),
>   nfa_advance_end(), and nfa_advance_var() with per-branch
>   early termination
> - Add tests in rpr_base.sql, rpr_nfa.sql, and rpr.sql covering
>   basic reluctant semantics, quantifier boundaries, interaction
>   with greedy quantifiers, and nested/alternation combinations

This is really good news!

>> I found reluctant quantifiers are useful but also I don't want to make
>> patch sets far bigger. How do you estimate the difficulty and the size
>> of the code for reluctant quantifiers?
> 
> You asked earlier how I estimated the difficulty of reluctant
> quantifiers.  It turned out the answer was subtraction, not
> addition -- removing altPriority and simplifying the match logic
> first made reluctant quantifiers almost trivial to add on top.

Glad to hear that you found simpler solution to implement reluchtant
quantifiers.

> Next I plan to work on the remaining FIXME (cycle prevention for
> patterns like (A*)*), A{0} implementation, and test reorganization.

Ok. Looking forward to the next patches.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  226. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-24T01:28:19Z

    Hi Tatsuo,

Here are six incremental patches on top of v43.

nocfbot-0001 through nocfbot-0004 are the same patches from the
previous round (32-bit test fix, PREV/NEXT restriction, ALT lexical
ordering, reluctant quantifiers).

nocfbot-0005: Detect zero-consumption NFA cycles

Adds a per-element visited bitmap to detect zero-consumption cycles
during DFS epsilon expansion.  Before each state's DFS, the bitmap
is cleared; as nfa_advance_state() recurses through epsilon
transitions, each elemIdx is marked visited.  If the same elemIdx
is reached again within the same DFS, it means an epsilon-only
loop -- the state is freed immediately.

The ad-hoc (count == 0 && min == 0) exit condition in
nfa_advance_end() is removed.  The FIXME test cases are resolved
and renamed to "Zero-Consumption Cycle Detection".

nocfbot-0006: Allow A{0} quantifier

With cycle detection in place, this becomes straightforward.
Lowers the {n} bound minimum from 1 to 0.  A{0} is treated as an
epsilon transition -- the variable is skipped entirely.

Next I plan to work on test reorganization, cross-database result
comparison, and a review pass over the NFA executor.

Best regards,
Henson

  227. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-24T02:56:25Z

    Hi Henson,

> Hi Tatsuo,
> 
> Here are six incremental patches on top of v43.
> 
> nocfbot-0001 through nocfbot-0004 are the same patches from the
> previous round (32-bit test fix, PREV/NEXT restriction, ALT lexical
> ordering, reluctant quantifiers).
> 
> nocfbot-0005: Detect zero-consumption NFA cycles
> 
> Adds a per-element visited bitmap to detect zero-consumption cycles
> during DFS epsilon expansion.  Before each state's DFS, the bitmap
> is cleared; as nfa_advance_state() recurses through epsilon
> transitions, each elemIdx is marked visited.  If the same elemIdx
> is reached again within the same DFS, it means an epsilon-only
> loop -- the state is freed immediately.
> 
> The ad-hoc (count == 0 && min == 0) exit condition in
> nfa_advance_end() is removed.  The FIXME test cases are resolved
> and renamed to "Zero-Consumption Cycle Detection".
> 
> nocfbot-0006: Allow A{0} quantifier
> 
> With cycle detection in place, this becomes straightforward.
> Lowers the {n} bound minimum from 1 to 0.  A{0} is treated as an
> epsilon transition -- the variable is skipped entirely.

Great!

> Next I plan to work on test reorganization, cross-database result
> comparison, and a review pass over the NFA executor.

Looking forward to seeing next patches.

BTW, in create_windowagg_plan (createplan.c),
around:
/* Build RPR pattern and filter defineClause */

collectPatternVariables, filterDefineClause and buildRPRPattern are
called in a block without any if or any other conditional
statements. This is an unusual codiing style in PostgreSQL.  I suggest
to fix this.  Attached is a proposed patch for this.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  228. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-24T04:38:32Z

    Hi Tatsuo,

> BTW, in create_windowagg_plan (createplan.c),
> around:
> /* Build RPR pattern and filter defineClause */
>
> collectPatternVariables, filterDefineClause and buildRPRPattern are
> called in a block without any if or any other conditional
> statements. This is an unusual codiing style in PostgreSQL.  I suggest
> to fix this.  Attached is a proposed patch for this.

Good catch, thank you!  I've renumbered your patch as nocfbot-0007
and extended the same cleanup to rpr.c and parse_rpr.c as
nocfbot-0008.

nocfbot-0007: Refactor create_windowagg_plan to remove bare
              variable-scoping block (your patch, renumbered)

nocfbot-0008: Remove bare variable-scoping blocks in RPR code

  Applies the same cleanup to rpr.c and parse_rpr.c, with minor
  pgindent formatting fixes.

I'll keep this coding style point in mind for future code as well.

Best regards,
Henson

  229. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-24T05:09:27Z

    Hi Henson,

>> BTW, in create_windowagg_plan (createplan.c),
>> around:
>> /* Build RPR pattern and filter defineClause */
>>
>> collectPatternVariables, filterDefineClause and buildRPRPattern are
>> called in a block without any if or any other conditional
>> statements. This is an unusual codiing style in PostgreSQL.  I suggest
>> to fix this.  Attached is a proposed patch for this.
> 
> Good catch, thank you!  I've renumbered your patch as nocfbot-0007
> and extended the same cleanup to rpr.c and parse_rpr.c as
> nocfbot-0008.
> 
> nocfbot-0007: Refactor create_windowagg_plan to remove bare
>               variable-scoping block (your patch, renumbered)
> 
> nocfbot-0008: Remove bare variable-scoping blocks in RPR code
> 
>   Applies the same cleanup to rpr.c and parse_rpr.c, with minor
>   pgindent formatting fixes.
> 
> I'll keep this coding style point in mind for future code as well.

Thank you for quick response!

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  230. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-24T11:44:17Z

    Hi Henson,

Currently we do not account the cost of RPR while planning.  Attached
is the first attempt to try to estimate the RPR costs. The cost model
is very simple:

expression cost per PATTERN variable * number of input tuples

Any idea to make this estimation better?

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  231. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-25T07:57:01Z

    Hi Tatsuo,

I recently purchased the ISO/IEC 19075-5:2021 standard document
through my company, and have started cross-referencing it against
the implementation.

> nocfbot-0005: Detect zero-consumption NFA cycles


The cycle detection itself is necessary, but I found a behavioral
difference from the standard (Section 7.2.8).

When a group body is nullable (e.g. A? in (A?){2,3}), the visited
bitmap blocks re-entry to the skipped variable, so the END element
never gets a chance to produce an exit state for count < min.
The standard/Perl allows empty iterations to count toward min.

I plan to fix this by adding a compile-time flag on END elements
that indicates the group body can iterate with empty matches.
When the flag is set, nfa_advance_end() will generate the exit
state even when count < min.

> nocfbot-0006: Allow A{0} quantifier
>

After reviewing the standard, I'd like to withdraw this patch.

Section 4.14.1 explicitly requires n > 0 for the {n} quantifier.
While A{0} works correctly as an epsilon transition with the cycle
detection in place, it violates the standard.

Since we aim for standard conformance, I think we should keep the
n >= 1 restriction for {n}.

I'll include this revert in the next patch set.

Best regards,
Choi

  232. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-25T10:16:50Z

    Hi Henson,

> Hi Tatsuo,
> 
> I recently purchased the ISO/IEC 19075-5:2021 standard document
> through my company, and have started cross-referencing it against
> the implementation.
> 
>> nocfbot-0005: Detect zero-consumption NFA cycles
> 
> 
> The cycle detection itself is necessary, but I found a behavioral
> difference from the standard (Section 7.2.8).
> 
> When a group body is nullable (e.g. A? in (A?){2,3}), the visited
> bitmap blocks re-entry to the skipped variable, so the END element
> never gets a chance to produce an exit state for count < min.
> The standard/Perl allows empty iterations to count toward min.
> 
> I plan to fix this by adding a compile-time flag on END elements
> that indicates the group body can iterate with empty matches.
> When the flag is set, nfa_advance_end() will generate the exit
> state even when count < min.

Ok.

>> nocfbot-0006: Allow A{0} quantifier
>>
> 
> After reviewing the standard, I'd like to withdraw this patch.
> 
> Section 4.14.1 explicitly requires n > 0 for the {n} quantifier.
> While A{0} works correctly as an epsilon transition with the cycle
> detection in place, it violates the standard.
> 
> Since we aim for standard conformance, I think we should keep the
> n >= 1 restriction for {n}.

This is my bad. I didn't realize the standard prohibits A{0} even I
have the standard document.

> I'll include this revert in the next patch set.

Thanks!
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  233. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-26T13:34:03Z

    Hi Tatsuo,

Here are ten incremental patches on top of v43.

nocfbot-0001 through nocfbot-0005 are the same as the previous round
(32-bit test fix, PREV/NEXT restriction, ALT lexical ordering,
reluctant quantifiers, cycle detection).

As discussed, the A{0} patch (old nocfbot-0006) is withdrawn since
Section 4.14.1 requires n > 0.  Your bare-block refactoring patches
are renumbered accordingly:

nocfbot-0006: Refactor create_windowagg_plan to remove bare
              variable-scoping block (your patch, renumbered)

nocfbot-0007: Remove bare variable-scoping blocks in RPR code

Three new patches:

nocfbot-0008: Fix empty-match iteration for nullable group bodies

This is the fix I mentioned for the Section 7.2.8 behavioral
difference.  When a group body is nullable (e.g. A? in (A?){2,3}),
the visited bitmap blocks re-entry to the skipped variable, so the
END element never gets a chance to produce an exit state for
count < min.

The fix adds a compile-time canEmptyLoop flag on END elements.
The fillRPRPattern functions now return whether each sub-pattern
is nullable, and this information propagates up to set the flag
on the enclosing END element.

At runtime, when the flag is set and count < min,
nfa_advance_end() creates a fast-forward exit state that jumps
directly past the group, treating all remaining required
iterations as empty.  This avoids the cycle detection while
still counting empty iterations toward min, matching the
standard and Perl behavior.

nocfbot-0009: Fix use-after-free in NFA alternation and
              optional-VAR routing

Two use-after-free bugs in the NFA advance phase:

1) nfa_advance_alt() reused the original state for the first ALT
   branch, then created copies for subsequent branches.  If
   nfa_advance_state() freed the original (e.g. via
   nfa_add_state_unique() deduplication), subsequent branches
   would copy from freed memory.  Fix: create independent states
   for all branches and free the original.

2) nfa_route_to_elem() created the skip state for optional VARs
   after calling nfa_add_state_unique(), which may free the
   source state.  Fix: create the skip state before add_unique.

nocfbot-0010: Remove redundant conditions and fix comments in
              NFA executor

Cleanup pass over NFA executor functions:

- nfa_states_equal(): removed always-true compareDepth > 0
  guard (RPRDepth is uint8, so depth + 1 is always >= 1)
- nfa_start_context(): removed always-true NULL/count guards
  and bare variable-scoping block
- nfa_advance_var(): removed always-true count > 0 checks in
  three places (nfa_advance_var is only reached after match
  phase where count++ already executed, and max=0 is forbidden)
- nfa_process_row(): fixed missing periods in phase comments

Best regards,
Henson

  234. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-26T13:46:01Z

    Hi hackers,

I'd like to share an internal design document for the NFA executor
in the row pattern recognition (RPR) patch.

The attached guide covers:

- Pattern element structure (RPRPatternElement fields and flags)
- Compilation phases (AST to flat element array)
- Runtime execution model (Match / Absorb / Advance phases)
- Cycle detection for zero-consumption loops
- Empty-loop handling for nullable group bodies
- Reluctant quantifier support
- Absorption optimization for redundant context pruning

The full English text follows below.  All three language
versions are also attached:

  RPR_NFA_Algorithm_Guide.txt     (English)
  RPR_NFA_Algorithm_Guide_KO.txt  (Korean)
  RPR_NFA_Algorithm_Guide_JA.txt  (Japanese)

These are reference documents intended to help reviewers understand
the NFA executor internals.  They are not part of the patch itself.

================================================================================
  PostgreSQL Row Pattern Recognition: Flat-Array Stream NFA Guide
================================================================================

  Target audience: Developers with a basic understanding of the PostgreSQL
                   executor and planner architecture

  Scope: The entire process from PATTERN/DEFINE clause parsing to NFA
         runtime execution

  Related code:
    - src/backend/parser/parse_rpr.c          (parser phase)
    - src/backend/optimizer/plan/rpr.c        (optimizer phase)
    - src/backend/executor/nodeWindowAgg.c    (executor phase)
    - src/include/nodes/plannodes.h           (plan node definitions)
    - src/include/nodes/execnodes.h           (execution state definitions)
    - src/include/optimizer/rpr.h             (types and constants)

================================================================================


What is a Flat-Array Stream NFA?

  The NFA in this implementation is not a traditional state-transition graph
  but a flat array of fixed-size 16-byte elements. At runtime, it processes
  the row stream in a forward-only manner, expanding epsilon transitions
  eagerly without backtracking.

  - Flat-Array: Pattern compiled into a flat array,
                not a graph (Chapter IV)
  - Stream:     Rows consumed sequentially in one direction,
                never revisited (Chapter XII)
  - NFA:        Nondeterministic execution where multiple states
                coexist within a single context (Chapter VI)


Chapter I  Row Pattern Recognition Overview
================================================================================

Row Pattern Recognition (hereafter RPR) is a feature introduced in SQL:2016
that matches regex-based patterns against ordered row sets.

The SQL standard defines two forms:

  Feature R010: MATCH_RECOGNIZE (FROM clause)
    - Dedicated table operator
    - Provides dedicated functions such as MATCH_NUMBER(), CLASSIFIER()
    - Supports ONE ROW PER MATCH / ALL ROWS PER MATCH

  Feature R020: RPR in a window (WINDOW clause)
    - Integrated into the existing window function framework
    - Supports ALL ROWS PER MATCH only
    - No MATCH_NUMBER()

This implementation targets Feature R020.

The basic syntax is as follows:

  SELECT ...
  OVER (
    PARTITION BY ...
    ORDER BY ...
    ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
    [INITIAL | SEEK]   -- SEEK is defined in the standard but not
implemented
    AFTER MATCH SKIP TO NEXT ROW | SKIP PAST LAST ROW
    PATTERN ( <regex> )
    DEFINE <variable> AS <condition>, ...
  )

The PATTERN clause is a regular expression over row pattern variables.
The DEFINE clause specifies boolean conditions that determine whether each
variable evaluates to true for the current row.

Example:

  PATTERN (A+ B)
  DEFINE A AS price > PREV(price),
         B AS price < PREV(price)

This pattern matches "a span where prices rise consecutively then drop."


Chapter II  Overall Processing Pipeline
================================================================================

RPR processing is divided into three phases:

  +------------------------------------------------------------+
  |  1. Parsing (Parser)                                       |
  |     SQL text -> PATTERN AST + DEFINE expression tree       |
  |                                                            |
  |  2. Compilation (Optimizer/Planner)                        |
  |     PATTERN AST -> optimization -> flat NFA element array  |
  |                                                            |
  |  3. Execution (Executor)                                   |
  |     Row-by-row matching via NFA simulation                 |
  +------------------------------------------------------------+

Each phase uses independent data structures, and the interfaces between
phases are well-defined:

  Parser -> Planner:    WindowClause.rpPattern (RPRPatternNode tree)
                        WindowClause.defineClause (TargetEntry list)

  Planner -> Executor:  WindowAgg.rpPattern (RPRPattern struct)
                        WindowAgg.defineClause (TargetEntry list)


Chapter III  Parsing Phase
================================================================================

III-1. Entry Point

  transformWindowDefinitions() (parse_clause.c)
    +-- transformRPR() (parse_rpr.c)

transformRPR() is invoked when RPCommonSyntax is present and performs the
following:

  (1) Frame option validation
      - Only ROWS is allowed (RANGE, GROUPS are not)
      - The start boundary must be CURRENT ROW
      - EXCLUDE option is not allowed

  (2) Transcription to WindowClause
      - Copies rpPattern, rpSkipTo, initial fields

  (3) DEFINE clause transformation (transformDefineClause)

III-2. PATTERN AST

The parser transforms the PATTERN clause into an RPRPatternNode tree.
Each node has one of the following four types:

  RPR_PATTERN_VAR    Variable reference. Name stored in varName field.
  RPR_PATTERN_SEQ    Concatenation. Children node list in children.
  RPR_PATTERN_ALT    Alternation. Branch node list in children.
  RPR_PATTERN_GROUP  Group (parentheses). Body node list in children.

All nodes have min/max fields to express quantifiers:

  A       -> VAR(A, min=1, max=1)
  A+      -> VAR(A, min=1, max=INF)
  A*      -> VAR(A, min=0, max=INF)
  A?      -> VAR(A, min=0, max=1)
  A{3,5}  -> VAR(A, min=3, max=5)

If the reluctant field is not -1, the quantifier is reluctant (non-greedy).

Example: PATTERN ((A+ B) | C*)

  ALT
  +-- SEQ
  |   +-- VAR(A, 1, INF)
  |   +-- VAR(B, 1, 1)
  +-- VAR(C, 0, INF)

III-3. DEFINE Clause Transformation

transformDefineClause() processes each DEFINE variable as follows:

  (1) Checks for duplicate variable names
  (2) Transforms the expression into a standard SQL expression
  (3) Coerces to Boolean type (coerce_to_boolean)
  (4) Wraps in a TargetEntry with the variable name set in resname

Variables that are used in PATTERN but not defined in DEFINE are implicitly
evaluated as TRUE (matching all rows).


Chapter IV  Compilation Phase
================================================================================

IV-1. Entry Point

  create_windowagg_plan() (createplan.c)
    +-- collectPatternVariables()   Collect variable names
    +-- filterDefineClause()        Remove unused DEFINE entries
    +-- buildRPRPattern()           NFA compilation (6 phases)

IV-2. The 6 Phases of buildRPRPattern()

  Phase 1: AST optimization (optimizeRPRPattern)
  Phase 2: Statistics collection (scanRPRPattern)
  Phase 3: Memory allocation (allocateRPRPattern)
  Phase 4: NFA element fill (fillRPRPattern)
  Phase 5: Finalization (finalizeRPRPattern)
  Phase 6: Absorbability analysis (computeAbsorbability)

IV-3. Phase 1: AST Optimization

After copying the parser-generated AST, the following optimizations are
applied:

  (a) SEQ flattening: Unwrap nested SEQ nodes
      SEQ(A, SEQ(B, C)) -> SEQ(A, B, C)

  (b) Consecutive variable merging: Merge consecutive occurrences of the
      same variable into a single quantifier
      A A -> A{2}
      A{2,3} A{1,2} -> A{3,5}

  (c) Consecutive group merging: Merge repeated identical groups
      (A B)+ (A B)+ -> (A B){2,INF}

  (d) Consecutive ALT merging: Merge repeated identical ALT nodes
      (A | B) (A | B) (A | B) -> (A | B){3}

  (e) Prefix/suffix absorption: Absorb identical sequences before/after
      a group
      A B (A B)+ -> (A B){2,INF}

  (f) ALT flattening and deduplication
      (A | (B | C)) -> (A | B | C)
      (A | B | A) -> (A | B)

  (g) Quantifier multiplication: Collapse nested quantifiers when safe
      (A+)+ -> A+
      (A{2,3}){5} -> A{10,15}

  (h) Single-child unwrap
      SEQ(A) -> A,  (A){1,1} -> A

IV-4. Phase 4: NFA Element Array Generation

Transforms the optimized AST into a flat array of RPRPatternElement.
This is the core data structure used for NFA simulation at runtime.

RPRPatternElement struct (16 bytes):

  Field      Size     Description
  ---------------------------------------------------------
  varId      1B      Variable ID (0-251) or control code (252-255)
  depth      1B      Group nesting depth
  flags      1B      Bit flags (see below)
  reserved   1B      Padding
  min        4B      Quantifier lower bound
  max        4B      Quantifier upper bound
  next       2B      Next element index (sequential flow)
  jump       2B      Branch target index (for ALT/GROUP)

Control codes:

  RPR_VARID_BEGIN (252)  Group start marker
  RPR_VARID_END   (253)  Group end marker
  RPR_VARID_ALT   (254)  Alternation start marker
  RPR_VARID_FIN   (255)  Pattern completion marker

Element flags (1 byte, bitmask):

  Bit  Constant                    Set on        Description
  --------------------------------------------------------------------------
  0x01 RPR_ELEM_RELUCTANT          VAR, BEGIN,   Non-greedy quantifier.
                                   END           Prefers shorter match: try
                                                 exit-loop first, then
repeat.
                                                 Set on VAR for simple
(A+?),
                                                 on BEGIN+END for group
((...)+?)

  0x02 RPR_ELEM_EMPTY_LOOP         END           Group body can produce
empty
                                                 match (all children
nullable).
                                                 Creates a fast-forward
exit clone
                                                 alongside the normal
loop-back so
                                                 cycle detection doesn't
kill
                                                 legitimate matches. (IV-4b)

  0x04 RPR_ELEM_ABSORBABLE_BRANCH  VAR, BEGIN,   Element lies within an
                                   END, ALT      absorbable region. Used at
                                                 runtime to track whether
the
                                                 current NFA state is in an
                                                 absorbable context.

  0x08 RPR_ELEM_ABSORBABLE         VAR, END      Absorption judgment point.
                                                 WHERE to compare
consecutive
                                                 iterations for absorption.
                                                 - Simple unbounded VAR
(A+):
                                                   set on the VAR itself
                                                 - Unbounded GROUP ((A B)+):
                                                   set on the END element
only

  Accessor macros:
    RPRElemIsReluctant(e)        (e)->flags & 0x01
    RPRElemCanEmptyLoop(e)       (e)->flags & 0x02
    RPRElemIsAbsorbableBranch(e) (e)->flags & 0x04
    RPRElemIsAbsorbable(e)       (e)->flags & 0x08

Example: PATTERN (A+ B | C)

  AST: ALT(SEQ(VAR(A,1,INF), VAR(B,1,1)), VAR(C,1,1))

  Compilation result:

  idx  varId  depth  min  max  next  jump  Description
  ---------------------------------------------------
   0   ALT    0      1    1    1     -1    Alternation start
   1   A(0)   1      1    INF  2     3     Branch 1: A+
   2   B(1)   1      1    1    4     -1    Branch 1: B -> FIN
   3   C(2)   1      1    1    4     -1    Branch 2: C -> FIN
   4   FIN    0      1    1    -1    -1    Pattern completion

  - idx 0: ALT marker. next(=1) is the start of the first branch
  - idx 1: Variable A. next(=2) is B, jump(=3) is the start of the second
branch
  - idx 2: Variable B. next(=4) is FIN
  - idx 3: Variable C. next(=4) is FIN
  - idx 4: FIN marker. Match completion signal

Roles of next and jump:

  - next: The next element to move to "after consuming" the current element.
          For VAR, the next position after a successful match.
          For BEGIN/END, the next position inside/outside the group.

  - jump: The element to "skip to."
          In ALT, a jump from one branch to the next branch.
          In BEGIN, a skip path to END+1 (for groups with min=0).
          In END, a loop-back to the start of the group body.

Example: PATTERN ((A B)+)

  idx  varId    depth  min  max  next  jump  Description
  ---------------------------------------------------
   0   BEGIN    0      1    INF  1     4     Group start
   1   A(0)     1      1    1    2     -1    A
   2   B(1)     1      1    1    3     -1    B
   3   END      0      1    INF  4     1     Group end
   4   FIN      0      1    1    -1    -1    Pattern completion

  - idx 0: BEGIN. next(=1) enters the group body.
           jump(=4) skips to after END = FIN (used when min=0).
  - idx 3: END. next(=4) exits the group.
           jump(=1) loops back to the start of the group body.

IV-4a. Reluctant Flag (RPR_ELEM_RELUCTANT)

The reluctant flag is set during Phase 4 (fillRPRPattern) when the AST node
has a non-negative reluctant field (reluctant >= 0). It reverses the
priority
of quantifier expansion at runtime:

  Greedy (default):  try loop-back first, then exit  (prefer longer match)
  Reluctant:         try exit first, then loop-back   (prefer shorter match)

The flag is set on all elements that carry the quantifier:

  Simple VAR (A+?):     RPR_ELEM_RELUCTANT on the VAR element
  Group ((...)+?):      RPR_ELEM_RELUCTANT on BEGIN and END elements

At runtime (nfa_advance), the flag controls DFS exploration order:

  VAR with quantifier:
    Greedy:    primary path = next (continue), clone = jump (skip)
    Reluctant: primary path = jump (skip), clone = next (continue)

  END element:
    Greedy:    primary path = jump (loop-back), clone = next (exit)
    Reluctant: primary path = next (exit), clone = jump (loop-back)

  BEGIN with min=0:
    Greedy:    primary path = next (enter group), clone = jump (skip)
    Reluctant: primary path = jump (skip), clone = next (enter group)

The absorption optimization requires greedy quantifiers. Reluctant
quantifiers are excluded from absorbability analysis (see IV-5).

IV-4b. Empty Loop Flag (RPR_ELEM_EMPTY_LOOP)

The empty-loop flag is set during Phase 4 (fillRPRPatternGroup) on the END
element when the group body is nullable -- i.e., every path through the
body can match zero rows (all children are nullable).

Example patterns that trigger this flag:

  (A?)*    A is nullable (min=0), so group body is nullable -> END gets flag
  (A? B?)+ Both children nullable -> body nullable -> END gets flag
  (A | B*) B* is nullable, making the ALT nullable -> END gets flag

The flag works in conjunction with the zero-consumption cycle detection
(elemIdx visited bitmap). Without this flag, cycle detection alone would
cause legitimate matches to fail.

Problem example: (A*){2,3}
  - Iteration 1: A* consumes all available rows -> count=1, END reached
  - Loop-back for iteration 2: A* matches zero rows -> END reached again
  - Cycle detection sees the same elemIdx on the same row -> state killed
  - count never reaches min(2) -> match fails (incorrect)

With the RPR_ELEM_EMPTY_LOOP flag, nfa_advance_end creates two paths:
the normal loop-back (which cycle detection will eventually kill) and
a fast-forward exit clone that bypasses the loop entirely.
(See IX-4(c) for detailed runtime behavior.)
    - Zero-consumption is impossible since body is not nullable

IV-5. Absorbability Analysis (RPR_ELEM_ABSORBABLE)

Context absorption is an optimization technique that reduces O(n^2) to O(n).
(Runtime behavior is described in Chapter VIII.)

This phase determines whether the pattern has a structure suitable for the
absorption optimization and sets flags on the relevant elements:

  RPR_ELEM_ABSORBABLE         Absorption comparison point
  RPR_ELEM_ABSORBABLE_BRANCH  Element within an absorbable region

Eligibility conditions:

  (1) SKIP PAST LAST ROW (not NEXT ROW)
  (2) Frame end is UNBOUNDED FOLLOWING

Structural conditions (isUnboundedStart + computeAbsorbabilityRecursive):

  Case 1: Simple VAR+ (e.g., A+)
          -> ABSORBABLE | ABSORBABLE_BRANCH set on the VAR
  Case 2: GROUP+ whose body consists only of {1,1} VARs (e.g., (A B)+)
          -> ABSORBABLE_BRANCH on children,
            ABSORBABLE | ABSORBABLE_BRANCH on END
  Case 3: GROUP+ whose body starts with VAR+ (e.g., (A+ B)+)
          -> Recurses from BEGIN into the body, applying Case 1.
            ABSORBABLE | ABSORBABLE_BRANCH set on A.
            B and END get no flags -> absorption stops once past A.

Absorbability is determined per-element, not per-pattern.
Absorption comparison is performed only when a state resides at an
element with the RPR_ELEM_ABSORBABLE flag. Once a state leaves the
flagged region, absorption is permanently disabled for that state.

Through this mechanism, the runtime guarantees monotonicity:
"a context that started earlier always subsumes a context that
started later."


Chapter V  NFA Runtime Data Structures
================================================================================

V-1. RPRNFAState -- NFA State

A single NFA state represents "how far the pattern has progressed."

  Field         Description
  ---------------------------------------------------------
  elemIdx       Index of the current pattern element
  counts[]      Repetition count per group depth
  isAbsorbable  Whether the state is in an absorbable region
  next          Next state in the linked list

The size of the counts array is (maxDepth + 1), allocated as a flexible
array member at the end of the struct.

Example: In PATTERN ((A B)+ C), a state waiting for B in the 3rd iteration

  Element array: [0:BEGIN(d0) 1:A(d1) 2:B(d1) 3:END(d0) 4:C(d0) 5:FIN]

  elemIdx = 2 (B, depth 1)
  counts[0] = 2 (depth 0: depth of END. Group completed 2 iterations)
  counts[1] = 1 (depth 1: depth of B. A matched in current iteration)

  Counts are indexed by depth, not by elemIdx.
  counts[0] is incremented when passing through END(depth 0),
  and the group repetition count is preserved even when
  the state is at B(depth 1).

Definition of two states being "equal":

  Two states are equal if they have the same elemIdx and the same counts
  up to the depth of that element.
  nfa_states_equal() compares counts[0..elem->depth] using memcmp.
  Only counts at or below the depth of the current element are meaningful.

V-2. RPRNFAContext -- Matching Context

A single context represents "a matching attempt started from a specific
start row."

  Field                 Description
  ---------------------------------------------------------
  states                Linked list of active NFA states
  matchStartRow         Row number where matching started
  matchEndRow           Row number where matching completed (-1 if
incomplete)
  lastProcessedRow      Last row processed
  matchedState          State that reached FIN (for greedy fallback)
  hasAbsorbableState    Whether this context can absorb other contexts
  allStatesAbsorbable   Whether this context can be absorbed
  next, prev            Doubly-linked list

Since the NFA is nondeterministic, multiple states can coexist
simultaneously within a single context.

Example: In PATTERN (A | B) C, if the first row matches both A and B,
two states coexist within the context:

  State 1: elemIdx=3 (waiting for C, via branch A)
  State 2: elemIdx=3 (waiting for C, via branch B)

In this case, since the (elemIdx, counts) of the two states are equal,
nfa_add_state_unique() retains only State 1 (branch A), which was
added first.
Because DFS processes the first branch of ALT first, the state via A
is registered first, and the state via B is discarded as a duplicate.
This is the preferment guarantee.

V-3. RPR Fields of WindowAggState

  nfaContext / nfaContextTail   Doubly-linked list of active contexts
  nfaContextFree                Reuse pool for contexts
  nfaStateFree                  Reuse pool for states
  nfaVarMatched                 Per-row cache: varMatched[varId]
  nfaVisitedElems               Bitmap for cycle detection
  nfaStateSize                  Precomputed size of RPRNFAState

Memory management:

  States and contexts are managed through their own free lists.
  Instead of palloc, they are obtained from the reuse pool, and
  returned to the pool upon deallocation.
  This reduces the overhead of frequent allocation/deallocation.


Chapter VI  NFA Execution: 3-Phase Model
================================================================================

VI-1. Entry Point and Overall Flow

When the window function processes each row, row_is_in_reduced_frame()
is called. This function determines whether the current row belongs to
a matched frame, and if necessary, calls update_reduced_frame() to
drive the NFA.

Flow of update_reduced_frame():

  (1) Find or create a context for the target row
  (2) Enter the row processing loop
  (3) After the loop ends, record the result in reduced_frame_map

Pseudocode of the row processing loop:

  targetCtx = nfa_get_head_context(pos)
  if targetCtx == NULL:
      targetCtx = nfa_start_context(pos)

  for currentPos = startPos ...
      if not nfa_evaluate_row(currentPos):  -- row does not exist
          nfa_finalize_all_contexts()       -- finalize all contexts
          break

      nfa_process_row(currentPos)           -- 3-phase processing
      nfa_start_context(currentPos + 1)     -- pre-create next start point
      nfa_cleanup_dead_contexts()           -- remove dead contexts

      if targetCtx->states == NULL:         -- target context exhausted
          break

Key point: Processing a single row may require processing multiple rows
ahead. Due to the nature of window functions, determining the frame for
row N requires looking at rows beyond N.

VI-2. Context Creation: nfa_start_context()

Creates a new context and performs the initial advance.

  (1) Allocate context via nfa_context_alloc()
  (2) Set matchStartRow = pos
  (3) Create initial state: elemIdx=0 (first pattern element), counts=all
zero
  (4) Call nfa_advance(initialAdvance=true)

The initial advance expands epsilon transitions at the beginning of
the pattern. For example, the initial advance for PATTERN ((A | B) C):

  Start: elemIdx=0 (ALT)
    -> Expand ALT branches
      -> elemIdx=1 (A) -- VAR, so add state; stop here
      -> elemIdx=2 (B) -- VAR, so add state; stop here

  Result: Two states in the context {waiting for A, waiting for B}

During the initial advance, reaching FIN is not recorded as a match.
This is to prevent empty matches (zero-length matches).

VI-3. Row Evaluation: nfa_evaluate_row()

Evaluates all variable conditions in the DEFINE clause at once for
the current row.

  for each defineClause[i]:
      result = ExecEvalExpr(defineClause[i])
      varMatched[i] = (not null and true)

To support row navigation operators such as PREV() and NEXT(),
the previous row, current row, and next row are set in separate slots:

  ecxt_scantuple  = previous row (for PREV reference)
  ecxt_outertuple = current row  (default reference)
  ecxt_innertuple = next row     (for NEXT reference)

The varMatched array is referenced later in Phase 1 (Match).

VI-4. nfa_process_row(): 3-Phase Processing

NFA processing for a single row is divided into three phases:

  +--------------------------------------------+
  |  Phase 1: MATCH (convergence)              |
  |  Compare the current row against each VAR  |
  |  state. Remove states that fail to match.  |
  |                                            |
  |  Phase 2: ABSORB (absorption)              |
  |  Merge duplicate contexts to prevent       |
  |  state explosion.                          |
  |                                            |
  |  Phase 3: ADVANCE (expansion)              |
  |  Expand epsilon transitions to prepare     |
  |  for the next row.                         |
  +--------------------------------------------+

This ordering is important:

  - Match executes first to "consume the current row."
  - Absorb executes immediately after Match, when states have been updated.
  - Advance executes last to prepare "states waiting for the next row."


Chapter VII  Phase 1: Match
================================================================================

nfa_match() iterates through each state in the context:

  (1) Check whether the state's elemIdx is a VAR element
  (2) Compare against the current row using nfa_eval_var_match()
  (3) Match success: increment repetition count, retain state
  (4) Match failure: remove state

Match determination (nfa_eval_var_match):

  If varId is within the range of defineVariableList:
      Use the value of varMatched[varId]

  If varId exceeds the range (variable not defined in DEFINE):
      Unconditionally true (matches all rows)

Immediate advance for simple VARs:

  For a VAR with min=1, max=1 where the next element is END,
  the Match phase processes through END immediately.
  This is necessary for accurate state comparison in Phase 2 (Absorb).

  Example: In PATTERN ((A B)+), when A matches, it immediately advances
  to B, and when B matches, it immediately advances through END to
  complete the group count. This enables absorption comparison with
  other contexts.


Chapter VIII  Phase 2: Absorb (Context Absorption)
================================================================================

VIII-1. Problem

In the current implementation, a new context is started for each row
processed.
Applying PATTERN (A+) to 10 rows produces 10 contexts,
each of which tracks state independently.

If there are N rows, the total number of states becomes O(N^2):

  Context 1 (started at row 1): can match A up to N times
  Context 2 (started at row 2): can match A up to N-1 times
  ...
  Context N (started at row N): can match A 1 time

VIII-2. Solution: Context Absorption

Key observation: a context started earlier contains
all matches of a later-started context (monotonicity principle).

If Context 1 started at row 1 and matched A 5 times,
the state where Context 2 (started at row 2) matched A 4 times
is already contained within Context 1.

Therefore Context 2 can be "absorbed" into Context 1.

VIII-3. Absorption Conditions

  (1) The pattern is marked as isAbsorbable (see IV-5)
  (2) allStatesAbsorbable of the target context is true
  (3) An earlier context "covers" all states of the target

Cover condition (nfa_states_covered):

  A state with the same elemIdx exists in the earlier context,
  and the count at that depth is greater than or equal -- then it is
covered.

VIII-4. Dual-Flag Design

Two boolean flags make the absorption decision efficient:

  hasAbsorbableState (monotonic: only true->false transition possible)
    "Does this context have the ability to absorb other contexts?"
    true if at least one absorbable state exists.
    Transitions to false when states are removed leaving no absorbable
states.
    Once false, it never becomes true again.

  allStatesAbsorbable (dynamic: can fluctuate)
    "Can this context be absorbed?"
    true if all states are in an absorbable region.
    Becomes false when a non-absorbable state is added; reverts to true
when it is removed.

VIII-5. Absorption Order

nfa_absorb_contexts() traverses from tail (newest) to head (oldest).

  for ctx = tail to head:
      if ctx.allStatesAbsorbable:
          for older = ctx.prev to head:
              if older.hasAbsorbableState:
                  if nfa_states_covered(older, ctx):
                      free(ctx)  -- absorbed
                      break

Since inspection starts from the newest context, the most recently started
(= having the shortest match) context is absorbed first.


Chapter IX  Phase 3: Advance (Epsilon Transition Expansion)
================================================================================

IX-1. Overview

nfa_advance() expands epsilon transitions from each state after Match,
generating "new states waiting for the next row."

An epsilon transition is a transition that moves without consuming a row:

  - ALT: branch to each alternative
  - BEGIN: enter group (or skip if min=0)
  - END: loop-back within group (or exit when condition is met)
  - FIN: record match completion
  - VAR loop/exit: repeat/exit according to the quantifier

Expansion stops upon reaching a VAR element, and the state is added.
This is because VAR is the element that "will consume the next row."

IX-2. Processing Order: DFS and Preferment

advance processes states in lexicographic order,
performing Depth-First Search (DFS) on each state.

This DFS order is what guarantees the SQL standard's "preferment":

  The branch that appears first in the PATTERN text takes precedence.

Example: PATTERN (A | B) C

  The first branch A of the ALT takes precedence over the second branch B.
  When both A and B can match, the match via A is selected.

nfa_add_state_unique() prevents duplicate addition of the same state,
so the state added first (= from the preferred branch) is retained.

IX-3. Routing Function: nfa_route_to_elem()

All inter-element transitions in the advance phase go through
nfa_route_to_elem().
This function branches its behavior based on the type of the next element:

  If the next element is VAR:
    (1) Add the state to the context (nfa_add_state_unique)
    (2) If the VAR has min=0, also add a skip path (recurse via next)
    -> Expansion stops here (VAR is the element that "will consume the next
row")

  If the next element is non-VAR (ALT, BEGIN, END, FIN):
    -> Recursively call nfa_advance_state() to continue expansion

With this structure, advance recursively follows epsilon transitions
until reaching a VAR, consistently stopping only at VAR elements.

IX-4. Per-Element advance Behavior

(a) ALT (nfa_advance_alt)

  Upon encountering an ALT element, all branches are expanded in order.
  The first element of each branch is connected via a jump pointer.

  idx=0 (ALT) -> branch 1 start (next) -> branch 2 start (jump) -> ...

  nfa_advance_state() is recursively called for each branch.

(b) BEGIN (nfa_advance_begin)

  Handles group entry.
  jump points to the element after END (= first element outside the group).

  Greedy (default):
    (1) Enter the group body (move via next, reset the count at that depth)
    (2) If min=0, also add a group skip path (move via jump)

  Reluctant:
    Order reversed -- skip path first, group entry second.
    If the skip path reaches FIN, the group entry path is not generated
    (shortest match preferred).

(c) END (nfa_advance_end)

  Handles group termination. This is the core of the repetition logic.

  Let count be the count at the current depth:

  count < min:
    Loop-back (move via jump, repeat the group body)

    If the RPR_ELEM_EMPTY_LOOP flag is set:
      In addition to loop-back, also add a fast-forward exit path.
      This is because the body may produce an empty match, causing count
      to never reach min. fast-forward resets counts[depth] to 0
      and exits via next (treating the remaining required iterations
      as empty matches).

  min <= count < max:
    Greedy: loop-back first, exit second
    Reluctant: exit first, loop-back second
              If the exit path reaches FIN, loop-back is omitted.

  count >= max:
    Unconditional exit (move via next)

  On exit: reset counts[depth] = 0, and if the next element is an outer END,
  increment the count at the outer depth.

(d) VAR (nfa_advance_var)

  Handles repeat/exit for a VAR element with a quantifier.

  Let count be the count at the current depth:

  count < min:
    Unconditional loop (stay at the same elemIdx, wait for the next row)

  min <= count < max:
    Greedy: loop first, exit (next) second
    Reluctant: exit first, loop second
              If the exit path reaches FIN, loop is omitted.

  count >= max:
    Unconditional exit (move via next)

  On exit: reset counts[depth] = 0.

(e) FIN

  Match success. The current state is moved to matchedState for recording,
  and matchEndRow is set to the current row.

  Upon reaching FIN, all remaining unprocessed states are removed
  (early termination). By DFS order, the path that reached FIN first
  has the highest preferment, so the rest are inferior paths.
  This is the core mechanism that guarantees preferment.

  In SKIP PAST LAST ROW mode, upon reaching FIN, subsequent contexts
  that started within the match range are immediately pruned.

IX-5. State Deduplication: nfa_add_state_unique()

When adding a new state to a context, it is compared against existing
states;
if an identical state already exists, it is not added.

Comparison criteria: elemIdx + counts[0..elem->depth] (see V-1)

This deduplication is the core mechanism that suppresses NFA state
explosion.
Because DFS order causes preferred-branch states to be added first,
identical states from lower-priority branches are automatically discarded.

IX-6. Cycle Detection: nfaVisitedElems

When a group body can produce an empty match (zero consumption),
looping back from END may cause an infinite loop.

Example: PATTERN ((A?)*)

  A? has min=0, so it can pass through without matching.
  If the outer group repeats: BEGIN -> A? skip -> END -> BEGIN -> ...

To prevent this:

  (1) At compile time: set the RPR_ELEM_EMPTY_LOOP flag on the END
      of groups whose body is nullable.
      The runtime effect of this flag is described in IX-4(c):
      when count < min, a fast-forward exit path is added,
      resolving the deadlock where count cannot increase due to empty
matches.

  (2) At runtime: initialize the nfaVisitedElems bitmap immediately before
      DFS expansion of each state within advance (once per state).
      During DFS, set the corresponding elemIdx bit when visiting each
element.
      If a previously visited elemIdx is revisited, that path is terminated.

  Note: the bitmap tracks only elemIdx and does not consider counts.
  Therefore, legitimate revisits to the same elemIdx but with different
counts
  may also be blocked. This only occurs when the group body is nullable
  (all paths can match empty), causing END -> loop-back -> skip -> END
  within a single DFS. In such cases the END element has the
  RPR_ELEM_EMPTY_LOOP flag, so the fast-forward exit (IX-4(c)) provides
  an alternative path that bypasses the cycle.


Chapter X  Match Result Processing
================================================================================

X-1. Reduced Frame Map

RPR match results are recorded in a byte array called reduced_frame_map.
One byte is allocated per row, and the value is one of the following:

  RF_NOT_DETERMINED (0)  Not yet processed
  RF_FRAME_HEAD     (1)  Start row of the match
  RF_SKIPPED        (2)  Interior row of the match (skipped in frame)
  RF_UNMATCHED      (3)  Match failure

The window function references this map to determine frame inclusion for
each row.

X-2. AFTER MATCH SKIP

Determines the starting point for the next match attempt after a successful
match:

  SKIP TO NEXT ROW:
    New match attempt begins from the row after the match start row.
    Overlapping matches are possible.

  SKIP PAST LAST ROW:
    New match attempt begins from the row after the match end row.
    Only non-overlapping matches are possible.

X-3. INITIAL vs SEEK

  Standard definition (section 6.12):
  INITIAL: "is used to look for a match whose first row is R."
  SEEK:    "is used to permit a search for the first match anywhere
           from R through the end of the full window frame."
  In either case, if there is no match, the reduced window frame is empty.
  The default is INITIAL.

  Current implementation:
  SEEK is not supported (the parser raises an error).
  Only INITIAL is supported, searching only for matches starting at each
  row position pos.


Chapter XI  Worked Example: Full Execution Trace
================================================================================

XI-1. Query

  SELECT company, tdate, price,
         first_value(price) OVER w AS start_price,
         last_value(price) OVER w AS end_price
  FROM stock
  WINDOW w AS (
    PARTITION BY company
    ORDER BY tdate
    ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
    AFTER MATCH SKIP PAST LAST ROW
    PATTERN (A+ B)
    DEFINE A AS price > PREV(price),
           B AS price < PREV(price)
  );

XI-2. Data

  Row#    tdate       price
  --------------------------
  0       2024-01-01  100
  1       2024-01-02  110
  2       2024-01-03  120
  3       2024-01-04  115
  4       2024-01-05  130

XI-3. Compilation Result

  PATTERN (A+ B) -> unchanged after optimization

  idx  varId  depth  min  max  next  jump
  ------------------------------------------
   0   A(0)   0      1    INF  1     -1     A+
   1   B(1)   0      1    1    2     -1     B
   2   FIN    0      1    1    -1    -1

  DEFINE: A -> "price > PREV(price)", B -> "price < PREV(price)"
  isAbsorbable = true (A+ is a simple unbounded VAR)

XI-4. Execution Trace

--- Row 0 (price=100) ---

  update_reduced_frame(0) called.

  Context C0 created (matchStartRow=0).
  Initial advance: elemIdx=0(A) -> VAR, so state is added.
  C0.states = [{elemIdx=0, counts=[0]}]

  nfa_evaluate_row(0):
    A: price(100) > PREV(price) -> no PREV -> false
    B: price(100) < PREV(price) -> no PREV -> false
    varMatched = [false, false]

  nfa_process_row(0):
    Phase 1 (Match): A(0) state vs varMatched[0]=false -> state removed
    C0.states = [] (empty)

    Phase 2 (Absorb): skipped (no states)
    Phase 3 (Advance): skipped (no states)

  C0.states is empty, so the loop terminates.
  matchEndRow < matchStartRow -> RF_UNMATCHED.
  register_reduced_frame_map(0, RF_UNMATCHED).

--- Row 1 (price=110) ---

  update_reduced_frame(1) called.

  Context C1 created (matchStartRow=1).
  Initial advance: C1.states = [{elemIdx=0, counts=[0]}]

  nfa_evaluate_row(1):
    A: 110 > PREV(100) -> true
    B: 110 < PREV(100) -> false
    varMatched = [true, false]

  nfa_process_row(1):
    Phase 1 (Match): A(0) match succeeds -> counts[0]++ -> counts=[1]
    C1.states = [{elemIdx=0, counts=[1]}]

    Phase 3 (Advance):
      State {elemIdx=0, counts=[1]}: A+ (min=1, count=1, max=INF)
        count >= min, so:
        Greedy -> loop first: keep {elemIdx=0, counts=[1]}
                  exit: reset counts[0]=0, next(=1) -> {elemIdx=1,
counts=[0]}
    C1.states = [{elemIdx=0, counts=[1]}, {elemIdx=1, counts=[0]}]

--- Row 2 (price=120) ---

  Context C2 created (matchStartRow=2).
  Initial advance: C2.states = [{elemIdx=0, counts=[0]}]

  nfa_evaluate_row(2):
    A: 120 > PREV(110) -> true
    B: 120 < PREV(110) -> false
    varMatched = [true, false]

  C1 nfa_process_row(2):
    Phase 1 (Match):
      {elemIdx=0, counts=[1]}: A matches -> counts=[2]
      {elemIdx=1, counts=[0]}: B does not match -> removed
    C1.states = [{elemIdx=0, counts=[2]}]

  C2 nfa_process_row(2):
    Phase 1 (Match):
      {elemIdx=0, counts=[0]}: A matches -> counts=[1]
    C2.states = [{elemIdx=0, counts=[1]}]

    Phase 2 (Absorb):
      Does C1 (started earlier) cover C2?
        C1: {elemIdx=0, counts=[2]}, C2: {elemIdx=0, counts=[1]}
        Same elemIdx, C1.counts >= C2.counts -> covered
      C2 absorbed. -> removed.

    Phase 3 (Advance):
      {elemIdx=0, counts=[2]}: Greedy -> loop + exit
        Loop: {elemIdx=0, counts=[2]}
        Exit: reset counts[0]=0, next(=1) -> {elemIdx=1, counts=[0]}
    C1.states = [{elemIdx=0, counts=[2]}, {elemIdx=1, counts=[0]}]

  Context C3 created (matchStartRow=3).

--- Row 3 (price=115) ---

  nfa_evaluate_row(3):
    A: 115 > PREV(120) -> false
    B: 115 < PREV(120) -> true
    varMatched = [false, true]

  nfa_process_row(3):
    Phase 1 (Match):
      {elemIdx=0, counts=[2]}: A does not match -> removed
      {elemIdx=1, counts=[0]}: B matches -> counts=[1]
    C1.states = [{elemIdx=1, counts=[1]}]

    Phase 3 (Advance):
      {elemIdx=1, counts=[1]}: B (min=1, max=1)
        count(1) >= max(1) -> unconditional exit
        Reset counts[0]=0, next = 2 (FIN)
      FIN reached -> matchEndRow = 3, matchedState recorded.
      Early termination: no remaining states, so completed immediately.
    C1.states = [] (empty after reaching FIN)

  C1.states is empty and matchEndRow=3 >= matchStartRow=1 -> match succeeds.

  register_reduced_frame_map(1, RF_FRAME_HEAD)
  register_reduced_frame_map(2, RF_SKIPPED)
  register_reduced_frame_map(3, RF_SKIPPED)

--- Row 4 (price=130) ---

  update_reduced_frame(4) called.
  C3 was already created but matchStartRow=3, so it is not applicable.
  New context C4 created (matchStartRow=4).

  nfa_evaluate_row(4):
    A: 130 > PREV(115) -> true
    B: 130 < PREV(115) -> false

  ... No subsequent rows, so nfa_finalize_all_contexts() is called.
  Match incomplete -> RF_UNMATCHED.

XI-5. Final Result

  Row 0: RF_UNMATCHED  -> frame = the row itself
  Row 1: RF_FRAME_HEAD -> frame = rows 1 through 3
  Row 2: RF_SKIPPED    -> inside row 1's match
  Row 3: RF_SKIPPED    -> inside row 1's match
  Row 4: RF_UNMATCHED  -> frame = the row itself


Chapter XII  Summary of Key Design Decisions
================================================================================

XII-1. Flat Array vs Tree-Based NFA

  Choice: Flat array (RPRPatternElement[])

  Rationale:
  - Cache-friendly: 16-byte fixed size, contiguous memory
  - Index-based references: 2-byte indices instead of pointers
  - Easy to serialize: can use memcpy when passing to plan nodes

XII-2. Forward-only Execution vs Backtracking

  Choice: Forward-only (state set tracking)

  Rationale:
  - Backtracking takes exponential time in the worst case
  - NFA simulation guarantees polynomial time
  - DFS order naturally guarantees preferment.
    Greedy/reluctant per quantifier requires only reversing the DFS order
  - Window functions receive sorted rows sequentially.
    Forward-only fits directly into this pipeline,
    whereas backtracking requires re-fetching previous rows
  - DEFINE conditions are SQL expressions (PREV, RUNNING aggregates, etc.)
    with high re-evaluation cost. Forward-only requires only one evaluation
    per row

XII-3. Per-Context Management

  Choice: Independent context per start row

  Rationale:
  - Supports overlapping matches under SKIP TO NEXT ROW
  - Determines the frame for each row independently
  - Absorption optimization can eliminate redundant contexts in O(n)

XII-4. Memory Pool Management

  Choice: Custom free list

  Rationale:
  - NFA states are created and destroyed in large numbers per row
  - Avoids palloc/pfree overhead
  - State size is variable (counts[] array), but within a single query
    maxDepth is fixed, so all states have the same size

XII-5. Execution Optimization Summary

  The following optimizations make the NFA simulation practical.

  -- Compile-time --

  (1) AST Optimization (IV-3)

    Simplifies the AST before converting the pattern to an NFA.
    Reduces the number of NFA elements through consecutive variable
    merging (A A -> A{2}), SEQ flattening, quantifier multiplication,
    and other transformations.

    Significance: Reducing the element count directly shrinks the state
    space, decreasing the cost of all subsequent runtime phases (match,
    absorb, advance).

  -- Runtime: advance phase --

  (2) Group Skip (IX-4(b))

    At the BEGIN of a group with min=0, uses jump to skip the entire
    group. Moves directly to the first element outside the group without
    exploring the group body. Greedy enters then skips; Reluctant skips
    then enters.

    Significance: For optional groups (min=0), immediately generates
    a skip path without exploring the body, avoiding unnecessary DFS
    expansion.

  (3) State Deduplication (IX-5)

    During advance, DFS may generate states with the same (elemIdx,
    counts) combination through multiple paths. Additionally, unlike
    VAR repetition, group repetition cannot perform absorption
    comparison using VAR states, so inline advance is performed from
    after Phase 1 match through to END; this process can also produce
    duplicate states reaching the same END.
    nfa_add_state_unique() blocks duplicate addition of identical states
    in both cases.

    Significance: Prevents exponential growth of the state count in
    ALT branches and quantifier expansion. Since DFS order causes the
    preferred branch's state to be registered first, identical states
    from lower-priority branches are automatically discarded, thereby
    also guaranteeing preferment.

  (4) Cycle Detection and Fast-Forward (IX-6, IX-4(c))

    When a nullable group body (e.g., A?) repeats empty matches,
    the END -> BEGIN loop-back can continue indefinitely.

    Two mechanisms resolve this:
    - A visited bitmap (nfaVisitedElems) blocks revisitation of the
      same element, preventing infinite loops (safety)
    - At an END with the RPR_ELEM_EMPTY_LOOP flag set, when
      count < min, the remaining required iterations are treated as
      empty matches and a fast-forward exit path out of the group is
      added (correctness)

    Significance: Cycle detection guarantees termination, and
    fast-forward guarantees that the min condition is satisfied.
    Without these, patterns containing nullable groups would fall
    into infinite loops or fail to match.

  (5) Match Pruning (IX-4(e))

    When a state reaches FIN during advance, all remaining unprocessed
    states of that context are removed. Because of DFS order, the path
    that reaches FIN first has the highest preferment, so the remaining
    paths are inferior.

    Significance: Once the best match is determined, exploration of
    inferior paths is immediately terminated. This mechanism achieves
    both preferment guarantees and performance optimization.

  -- Runtime: inter-context --

  (6) Early Termination (SKIP PAST LAST ROW)

    In SKIP PAST LAST ROW mode, when a match is found, subsequent
    contexts whose start rows fall within the match range are pruned
    immediately without further processing.
    In SKIP TO NEXT ROW mode, overlapping contexts are preserved
    because each row requires its own independent match.

    Significance: Prunes subsequent contexts whose start rows overlap
    with a prior match range, avoiding unnecessary processing.

  (7) Context Absorption (Chapter VIII)

    If an independent context is created for each row, O(n^2) states
    accumulate. By exploiting the monotonicity that an earlier-started
    context subsumes the states of a later-started context, redundant
    contexts are eliminated early.

    Absorbability is determined per-element; comparison is performed
    only at elements with the RPR_ELEM_ABSORBABLE flag (see IV-5).

    Significance: Keeps the number of active contexts at a constant
    level, achieving O(n^2) -> O(n) time complexity. Without this,
    performance degrades sharply on long partitions.


Appendix A. Key Function Index
================================================================================

  Function                       File                  Role
  ----------------------------------------------------------------------
  transformRPR                   parse_rpr.c           Parser entry point
  transformDefineClause          parse_rpr.c           DEFINE transformation
  collectPatternVariables        rpr.c                 Variable collection
  filterDefineClause             rpr.c                 DEFINE filtering
  buildRPRPattern                rpr.c                 NFA compilation main
  optimizeRPRPattern             rpr.c                 AST optimization
  fillRPRPattern                 rpr.c                 NFA element
generation
  finalizeRPRPattern             rpr.c                 Finalization
  computeAbsorbability           rpr.c                 Absorption analysis
  update_reduced_frame           nodeWindowAgg.c       Execution main loop
  nfa_start_context              nodeWindowAgg.c       Context creation
  nfa_process_row                nodeWindowAgg.c       3-phase processing
  nfa_evaluate_row               nodeWindowAgg.c       DEFINE evaluation
  nfa_match                      nodeWindowAgg.c       Phase 1
  nfa_absorb_contexts            nodeWindowAgg.c       Phase 2
  nfa_advance                    nodeWindowAgg.c       Phase 3
  nfa_advance_state              nodeWindowAgg.c       Per-state branching
  nfa_route_to_elem              nodeWindowAgg.c       Element routing
  nfa_advance_alt                nodeWindowAgg.c       ALT handling
  nfa_advance_begin              nodeWindowAgg.c       BEGIN handling
  nfa_advance_end                nodeWindowAgg.c       END handling
  nfa_advance_var                nodeWindowAgg.c       VAR handling
  nfa_add_state_unique           nodeWindowAgg.c       Deduplication
  nfa_states_covered             nodeWindowAgg.c       Absorption coverage
check


Appendix B. Data Structure Relationship Diagram
================================================================================

  Parser Layer
  --------
  RPCommonSyntax
    |--- rpSkipTo: RPSkipTo
    |--- initial: bool
    +--- rpPattern: RPRPatternNode* (tree)
         |--- nodeType: VAR | SEQ | ALT | GROUP
         |--- min, max: quantifier
         |--- varName: variable name (VAR only)
         +--- children: List* (SEQ/ALT/GROUP only)

  Planner Layer
  ----------
  WindowAgg (plan node)
    |--- rpSkipTo: RPSkipTo
    |--- defineClause: List<TargetEntry>
    +--- rpPattern: RPRPattern*
         |--- numVars: int
         |--- varNames: char**
         |--- maxDepth: RPRDepth
         |--- isAbsorbable: bool
         |--- numElements: int
         +--- elements: RPRPatternElement[]  (flat array)
              |--- varId      (1B)
              |--- depth      (1B)
              |--- flags      (1B)
              |--- reserved   (1B)
              |--- min, max   (4B + 4B)
              +--- next, jump (2B + 2B)

  Executor Layer
  ----------
  WindowAggState
    |--- rpPattern: RPRPattern* (copied from plan)
    |--- defineClauseList: List<ExprState>
    |--- nfaVarMatched: bool[] (per-row cache)
    |--- nfaVisitedElems: bitmapword* (cycle detection)
    |--- nfaContext <-> nfaContextTail (doubly-linked list)
    |   +--- RPRNFAContext
    |       |--- states: RPRNFAState* (linked list)
    |       |   |--- elemIdx
    |       |   |--- counts[]
    |       |   +--- isAbsorbable
    |       |--- matchStartRow, matchEndRow
    |       |--- matchedState (cloned on FIN arrival)
    |       |--- hasAbsorbableState
    |       +--- allStatesAbsorbable
    |--- nfaContextFree (recycling pool)
    +--- nfaStateFree (recycling pool)


Appendix C. NFA Element Array Examples
================================================================================

C-1. PATTERN (A B C)

  idx  varId  depth  min  max  next  jump
  ------------------------------------------
   0   A      0      1    1    1     -1
   1   B      0      1    1    2     -1
   2   C      0      1    1    3     -1
   3   FIN    0      1    1    -1    -1

C-2. PATTERN (A+ B*)

  idx  varId  depth  min  max  next  jump  flags
  -------------------------------------------------
   0   A      0      1    INF  1     -1    ABSORBABLE | ABSORBABLE_BRANCH
   1   B      0      0    INF  2     -1
   2   FIN    0      1    1    -1    -1

  Only A+ is the absorption point (Case 1). Once past A,
  absorption is permanently disabled for that state.

C-3. PATTERN (A | B | C)

  idx  varId  depth  min  max  next  jump
  ------------------------------------------
   0   ALT    0      1    1    1     -1     alternation start
   1   A      1      1    1    4     2      branch 1 -> FIN, jump -> branch
2
   2   B      1      1    1    4     3      branch 2 -> FIN, jump -> branch
3
   3   C      1      1    1    4     -1     branch 3 -> FIN
   4   FIN    0      1    1    -1    -1

C-4. PATTERN ((A B)+ C)

  idx  varId    depth  min  max  next  jump  flags
  -----------------------------------------------------------
   0   BEGIN    0      1    INF  1     4     ABSORBABLE_BRANCH
   1   A        1      1    1    2     -1    ABSORBABLE_BRANCH
   2   B        1      1    1    3     -1    ABSORBABLE_BRANCH
   3   END      0      1    INF  4     1     ABSORBABLE | ABSORBABLE_BRANCH
   4   C        0      1    1    5     -1
   5   FIN      0      1    1    -1    -1

  Case 2: GROUP+ with {1,1} body VARs. A, B are branches;
  END is the absorption point. Compare with C-6 (Case 3).

C-5. PATTERN ((A | B)+? C)

  idx  varId    depth  min  max   next  jump  flags
  ---------------------------------------------------
   0   BEGIN    0      1    INF   1     5     RELUCTANT, group start
   1   ALT      1      1    1     2     -1    alternation start
   2   A        2      1    1     4     3     branch 1
   3   B        2      1    1     4     -1    branch 2
   4   END      0      1    INF   5     1     RELUCTANT, group end
   5   C        0      1    1     6     -1
   6   FIN      0      1    1     -1    -1

C-6. PATTERN ((A+ B)+ C)  -- Absorbability flag example

  idx  varId    depth  min  max   next  jump  flags
  ----------------------------------------------------------
   0   BEGIN    0      1    INF   1     4     ABSORBABLE_BRANCH, group start
   1   A        1      1    INF   2     -1    ABSORBABLE | ABSORBABLE_BRANCH
   2   B        1      1    1     3     -1
   3   END      0      1    INF   4     1     group end
   4   C        0      1    1     5     -1
   5   FIN      0      1    1     -1    -1

  Recurses from BEGIN into the body -> A matches Case 1 (simple VAR+).
  A gets ABSORBABLE | ABSORBABLE_BRANCH, BEGIN gets ABSORBABLE_BRANCH.
  B and END get no flags -> absorption stops once the state advances to B.
  (See IV-5 Case 3)

C-7. PATTERN ((A+ B | C*)+ D)  -- Per-branch absorption in ALT

  idx  varId    depth  min  max   next  jump  flags
  ----------------------------------------------------------
   0   BEGIN    0      1    INF   1     6     ABSORBABLE_BRANCH
   1   ALT      1      1    1     2     -1    ABSORBABLE_BRANCH
   2   A        2      1    INF   3     4     ABSORBABLE | ABSORBABLE_BRANCH
   3   B        2      1    1     5     -1
   4   C        2      0    INF   5     -1    ABSORBABLE | ABSORBABLE_BRANCH
   5   END      0      1    INF   6     1     EMPTY_LOOP
   6   D        0      1    1     7     -1
   7   FIN      0      1    1     -1    -1

  ALT branches are checked independently for absorbability.
  Branch 1: A+ matches Case 1 -> A gets ABSORBABLE. B has no flag.
  Branch 2: C* matches Case 1 -> C gets ABSORBABLE.
  Both A and C get ABSORBABLE_BRANCH as part of their respective branch
paths.
  END has EMPTY_LOOP: branch 2 (C*) is nullable, making the group body
nullable.
  BEGIN and ALT get ABSORBABLE_BRANCH (on the path to absorbable elements).


================================================================================
  End of document
================================================================================

Best regards,
Henson

  235. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-27T02:37:33Z

    Hi Henson,

> Hi Tatsuo,
> 
> Here are ten incremental patches on top of v43.

Thanks.

BTW, I noticed that RPR accepts table name qualified variables in
DEFINE clause.

SELECT company, tdate, price, first_value(price) OVER w, last_value(price) OVER w,
 nth_value(tdate, 2) OVER w AS nth_second
 FROM stock
 WINDOW w AS (
 PARTITION BY company
 ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
 INITIAL
 PATTERN (START UP+ DOWN+)
 DEFINE
  START AS TRUE,
  UP AS stock.price > PREV(price),	<-- table name "stock" is uded
  DOWN AS price < PREV(price)
);

The standard does not allow to use range variables, declared in the
FROM clause, in the DEFINE clause (19075-5 6.5). Attached patch raises
an error in this case.

psql:a.sql:13: ERROR:  range var qualified name "stock.price" is not allowed in DEFINE clause
LINE 11:   UP AS stock.price > PREV(price),
                 ^

Also, currently we do not support pattern variable range vars in the
DEFINE caluse (e.g. UP.price). If used, we see a confusing error
message:

ERROR: missing FROM-clause entry for table "UP"
LINE 13: UP AS UP.price > PREV(price),
               ^

The attached patch errors out differently. I believe this is an
enhancement.

psql:a.sql:13: ERROR:  range var qualified name "up.price" is not allowed in DEFINE clause
LINE 11:   UP AS UP.price > PREV(price),
                 ^

Thoughts?
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  236. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-27T05:44:01Z

    Hi Tatsuo,

The standard does not allow to use range variables, declared in the
> FROM clause, in the DEFINE clause (19075-5 6.5). Attached patch raises
> an error in this case.
>

Agreed. §6.5 is explicit about mutual exclusivity of the two sets of
range variables. Your patch is correct for this case.

Also, currently we do not support pattern variable range vars in the
> DEFINE caluse (e.g. UP.price). If used, we see a confusing error
> message:
>
> ERROR: missing FROM-clause entry for table "UP"
> LINE 13: UP AS UP.price > PREV(price),
>                ^
>

Pattern variable qualified names like `UP.price` are actually valid
standard syntax (§4.16 uses them in examples), so "is not allowed"
is misleading — "is not supported" would be more accurate.

To distinguish the two cases, we could expose `patternVarNames` via
ParseState (it is already collected by `validateRPRPatternVarCount`
before DEFINE expressions are transformed) and check in
`transformColumnRef` whether the qualifier is a pattern variable:

  - pattern variable qualifier → "not supported"
  - anything else              → "not allowed"

Would it be okay if I revise the patch along those lines?

Best regards,
Henson

  237. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-27T05:55:39Z

    Hi Henson,

> The standard does not allow to use range variables, declared in the
>> FROM clause, in the DEFINE clause (19075-5 6.5). Attached patch raises
>> an error in this case.
>>
> 
> Agreed. §6.5 is explicit about mutual exclusivity of the two sets of
> range variables. Your patch is correct for this case.
> 
> Also, currently we do not support pattern variable range vars in the
>> DEFINE caluse (e.g. UP.price). If used, we see a confusing error
>> message:
>>
>> ERROR: missing FROM-clause entry for table "UP"
>> LINE 13: UP AS UP.price > PREV(price),
>>                ^
>>
> 
> Pattern variable qualified names like `UP.price` are actually valid
> standard syntax (§4.16 uses them in examples), so "is not allowed"
> is misleading ― "is not supported" would be more accurate.

Fair enough.

> To distinguish the two cases, we could expose `patternVarNames` via
> ParseState (it is already collected by `validateRPRPatternVarCount`
> before DEFINE expressions are transformed) and check in
> `transformColumnRef` whether the qualifier is a pattern variable:

Oh, I didn't realize it.

>   - pattern variable qualifier → "not supported"
>   - anything else              → "not allowed"
> 
> Would it be okay if I revise the patch along those lines?

Sure. Thanks.
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  238. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-27T11:30:57Z

    Hi Tatsuo,

Currently we do not account the cost of RPR while planning.  Attached
> is the first attempt to try to estimate the RPR costs. The cost model
> is very simple:
>
> expression cost per PATTERN variable * number of input tuples
>
> Any idea to make this estimation better?
>

 >   foreach(lc, windowFuncs)
>   {
>   ...
> + /* also add DEFINE clause expressions' cost to per-input-row costs */
> + if (winclause->rpPattern)
> + {
> + List   *pattern_vars; /* list of pattern variable names */
> + ListCell   *lc2;
> +
> + pattern_vars = collectPatternVariables(winclause->rpPattern);
> +
> + /* iterate according to the pattern variable */
> + foreach(lc2, pattern_vars)
> + {
> + char   *ptname = strVal((char *) lfirst(lc2));

`collectPatternVariables` returns a list of String nodes (via
`makeString()`), so `strVal(lfirst(lc2))` is the idiomatic form.
The `(char *)` cast is misleading.

There is also a correctness issue: DEFINE expressions belong to the
window clause, not to individual window functions, so their cost
should not be multiplied by the number of window functions sharing
the clause.

The fix is to compute the DEFINE cost once outside the loop and add
it to `startup_cost` and `total_cost` directly, after the
`foreach(lc, windowFuncs)` block.

Regarding the cost model: the NFA executor evaluates all DEFINE
expressions once per row into a shared `nfaVarMatched[]` array that
all active contexts read from, and contexts advance strictly forward
so no prior row is ever re-evaluated.  The one-evaluation-per-row
cost model is therefore accurate.  NFA-aware cost
modeling could be built on top of this foundation in a separate patch
down the road, once the NFA implementation has matured.

For now, the DEFINE expression costs themselves already serve as a
natural penalty — a window clause with RPR will consistently appear
more expensive than a comparable plain window function.  This gives
the surrounding plan a reasonable cost signal for decisions such as
join ordering and materialization of RPR subqueries.  So the current
approach is reasonable as a first step.

Other than that, the approach looks good to me.  Would it be okay if
I revise the patch along those lines?

Best regards,
Henson

  239. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-02-27T13:54:56Z

    Hi Henson,

> Hi Tatsuo,
> 
> Currently we do not account the cost of RPR while planning.  Attached
>> is the first attempt to try to estimate the RPR costs. The cost model
>> is very simple:
>>
>> expression cost per PATTERN variable * number of input tuples
>>
>> Any idea to make this estimation better?
>>
> 
>  >   foreach(lc, windowFuncs)
>>   {
>>   ...
>> + /* also add DEFINE clause expressions' cost to per-input-row costs */
>> + if (winclause->rpPattern)
>> + {
>> + List   *pattern_vars; /* list of pattern variable names */
>> + ListCell   *lc2;
>> +
>> + pattern_vars = collectPatternVariables(winclause->rpPattern);
>> +
>> + /* iterate according to the pattern variable */
>> + foreach(lc2, pattern_vars)
>> + {
>> + char   *ptname = strVal((char *) lfirst(lc2));
> 
> `collectPatternVariables` returns a list of String nodes (via
> `makeString()`), so `strVal(lfirst(lc2))` is the idiomatic form.
> The `(char *)` cast is misleading.

Ok.

> There is also a correctness issue: DEFINE expressions belong to the
> window clause, not to individual window functions, so their cost
> should not be multiplied by the number of window functions sharing
> the clause.

You are right.

> The fix is to compute the DEFINE cost once outside the loop and add
> it to `startup_cost` and `total_cost` directly, after the
> `foreach(lc, windowFuncs)` block.

Looks good.

> Regarding the cost model: the NFA executor evaluates all DEFINE
> expressions once per row into a shared `nfaVarMatched[]` array that
> all active contexts read from, and contexts advance strictly forward
> so no prior row is ever re-evaluated.  The one-evaluation-per-row
> cost model is therefore accurate.  NFA-aware cost
> modeling could be built on top of this foundation in a separate patch
> down the road, once the NFA implementation has matured.
> 
> For now, the DEFINE expression costs themselves already serve as a
> natural penalty ― a window clause with RPR will consistently appear
> more expensive than a comparable plain window function.  This gives
> the surrounding plan a reasonable cost signal for decisions such as
> join ordering and materialization of RPR subqueries.  So the current
> approach is reasonable as a first step.
> 
> Other than that, the approach looks good to me.  Would it be okay if
> I revise the patch along those lines?

Yes, no problem. Thanks!
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  240. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-02-28T16:59:56Z

    Hi hackers,

I'd like to propose an extension to the context absorption optimization
for Row Pattern Recognition in the window clause: PREFIX pattern
absorption. This refines the "anchored pattern absorption" concept I
posted earlier [1], renaming it to "PREFIX pattern absorption" per
Ishii-san's feedback [2], and formalizing the absorption conditions
with count-dominance analysis.

It builds on the existing absorption framework to handle patterns where
a fixed-length prefix precedes the unbounded greedy repetition, using a
"shadow path" design.

[1]
https://www.postgresql.org/message-id/CAAAe_zAEg7sVM=WDwXMyE-odGmQyXSVi5ZzWgye6SupSjdMKpg@mail.gmail.com
[2]
https://www.postgresql.org/message-id/20260217.180053.64368434967157940.ishii@postgresql.org


1. Problem Definition
=====================

1.1 Current Absorption Scope

Context absorption currently applies to patterns with an unbounded-start
structure -- where an unbounded greedy repetition appears at the very
beginning of the pattern.

Example: A+ B -- the first element A+ is an unbounded repetition, so an
older context always has a repeat count greater than or equal to any
newer context, making immediate absorption possible.

1.2 The PREFIX Pattern Problem

For patterns like START SECOND A+ B+, where a finite-length prefix
(PREFIX) precedes the unbounded repetition (A+), absorption does not
currently work.

Why:
- A new context starts at the START element, so it is not at the same
  pattern position as an older context in the A+ region.
- The allStatesAbsorbable condition is not satisfied, so absorption
  never fires.
- Contexts accumulate while traversing the PREFIX, causing O(n^2)
  regression.


2. Relationship to the Existing Framework
==========================================

2.1 Position in the Three-Phase Execution Model

The Match -> Absorb -> Advance model is preserved as-is. PREFIX
absorption is an extension of the Absorb phase's eligibility criteria,
not a new phase.

2.2 Prerequisites

The same four prerequisites as existing absorption apply:

1. Greedy quantifier -- A+ must be greedy.
2. SKIP TO PAST LAST ROW -- prior matches suppress subsequent start
   points.
3. UNBOUNDED FOLLOWING -- all contexts see the same future rows.
4. Match-start-independent DEFINE predicates -- including predicates of
   PREFIX elements.

2.3 Extension of the Dominance Order

In existing absorption, the dominance condition is:

    C_old and C_new are in the absorbable region of the same element,
    and C_old.counts >= C_new.counts.

In PREFIX patterns this condition cannot be directly satisfied: C_new is
in the PREFIX region while C_old is in the BODY region. We therefore
need a mechanism to track what C_new's state *would be* if it had
already reached the BODY region.


3. Design: Shadow Path
=======================

3.1 Core Idea

Each context logically executes two parallel paths:

- Original path: executes the full pattern in order (PREFIX -> BODY).
  This is the path that produces the actual match result and evaluates
  PREFIX predicates.

- Shadow path: skips the PREFIX and starts directly in the BODY region
  (A+).
  - Used solely for absorption eligibility decisions.
  - Does not produce match results.
  - Actually evaluates the BODY element's DEFINE predicates to track
    counts accurately.
  - The shadow is placed at the BODY start state upon creation and
    immediately advances, so its initial count is 1.
  - Not created if there is no preceding context (nothing to be
    absorbed into).
  - Discarded when the preceding context dies.
  - Discarded when the shadow leaves the absorbable region (BODY->TAIL
    transition or match failure).

3.2 Absorption Conditions

For C_old to absorb C_new, all of the following must hold:

1. C_old is past the PREFIX and in the absorbable region (BODY).
   - hasAbsorbableState = true (existing flag)

2. C_new was created at or after the row where C_old entered the
   absorbable region.
   - "Same row" means: in the Match phase C_old enters BODY and C_new
     is created; the subsequent Absorb phase of that same row can then
     perform the absorption check.

3. C_new's shadow path is alive in the absorbable region.
   - allStatesAbsorbable = true (evaluated on the shadow)

4. C_old.counts[depth] >= C_new.shadow.counts[depth]
   - Same count-dominance condition as existing absorption (equality
     counts as dominance).
   - The shadow's count increases as it processes rows, so the
     comparison uses actual counts at absorption time.

Condition 2 is the key insight. Only contexts created after C_old has
passed through the PREFIX into the absorbable region are eligible for
absorption. Contexts created while C_old is still in the PREFIX cannot
be absorbed.

3.3 Common Fate

Soundness argument for absorption:

    C_old's original path and C_new's shadow path are at the same BODY
    element (A+). By match-start independence, the DEFINE predicates of
    both paths produce identical results on the same row. If C_new's
    shadow can match at some future row r, then C_old can also complete
    a match at row r (it has matched at least as many times, sees the
    same future rows, and DEFINE evaluations are identical). Under SKIP
    TO PAST LAST ROW, C_old's match includes C_new's start point, so
    removing C_new does not lose any reported match.

3.4 Complexity

With PREFIX absorption, the maximum number of simultaneously active
contexts is bounded by PREFIX_length + 1. Every C_new created after
C_old enters BODY is absorbed immediately, so the only contexts that can
coexist are those still traversing the PREFIX plus the single C_old in
BODY. Since PREFIX length is a per-pattern constant, overall execution
complexity is O(n).

3.5 Removal of Non-Absorbable Contexts

Contexts that do not satisfy condition 2 -- i.e., those created before
C_old entered BODY -- cannot be removed by PREFIX absorption.

These contexts are removed by the existing SKIP TO PAST LAST ROW
mechanism, which is separate from PREFIX absorption. When C_old
successfully matches, all subsequent contexts whose start points fall
within C_old's match range are removed as overlaps. Since contexts
created during the PREFIX region have start points within C_old's match
range, they are automatically removed upon C_old's match completion.

The number of such contexts is at most PREFIX_length, which does not
affect the O(n) complexity (see Section 3.4).


4. Compile-Time Analysis
=========================

4.1 Pattern Structure Recognition

At compile time, traverse the pattern elements from the start until the
first unbounded greedy repetition is found. All preceding fixed-length
elements form the PREFIX; the unbounded element is the BODY; everything
after it is the TAIL. If the first element is already unbounded, the
pattern has no PREFIX and existing absorption applies directly.

    Pattern: E1 E2 ... Ek  A+          B+ ...
             \__________/  \__/         \___/
              PREFIX        BODY         TAIL
                           (absorbable)  (non-absorbable)


5. Worked Example
==================

Pattern: START SECOND A+ B+
PREFIX: START, SECOND (length 2)
BODY: A+ (absorbable)
TAIL: B+ (non-absorbable)

Data: r0-r3 satisfy A only, r4-r5 satisfy B only, r6 satisfies neither
A nor B.

Row  C1(orig)   C2(orig)   C2(shadow)  C3(orig)   C3(shadow)  Active
r0   START       -           -           -           -          1
r1   SECOND      START       A(cnt=1)    -           -          2
r2   A(cnt=1)    SECOND      A(cnt=2)    START       A(cnt=1)   3
     ^ C1 reaches BODY
     -> C3: shadow A(cnt=1), created at C1's BODY entry
        -> absorbed (1 >= 1)
     -> C2: created before C1's BODY entry
        -> not absorbable                                       2
r3   A(cnt=2)    A(cnt=1)    A(cnt=3)    -           -          2
r4   B(cnt=1)    B(cnt=1)    (discard)   -           -          2
     ^ TAIL      ^ TAIL      ^ shadow A+->B+ -> TAIL -> discard
r5   B(cnt=2)    B(cnt=2)    -           -           -          2
r6   neither A nor B -> B+ ends -> C1 match (r0-r5)             0
     C2 removed by SKIP TO PAST LAST ROW overlap
     (C2 start r1 falls within C1 match range r0-r5)

Max concurrent contexts: 3 = PREFIX_length(2) + 1

Key observations:
- C3 is immediately removed at r2 by PREFIX absorption (shadow-path-
  based).
- C2's shadow is discarded at r4 when A+->B+ transition moves it into
  TAIL.
- C2 is not absorbable (condition 2 not met), survives until r6, then
  removed by SKIP TO PAST LAST ROW overlap when C1 matches
  (Section 3.5).

This is a design proposal at this stage. Thoughts and feedback welcome.

Best regards,
Henson

  241. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-03-01T11:02:38Z

    Hi Tatsuo,

Attached are two more incremental patches (nocfbot-0011, nocfbot-0012)
on top of v43, continuing the nocfbot-0001..0010 series.

nocfbot-0011: Add RPR DEFINE expression cost to WindowAgg cost estimation

  Revised version of your cost estimation patch [1].  Fixes the
  `(char *)` cast and moves DEFINE cost outside the windowFuncs loop.

nocfbot-0012: Reject qualified column references in RPR DEFINE clause

  Revised version of your qualified column reference patch [2].
  Exposes `patternVarNames` via `ParseState.p_rpr_pattern_vars` to
  distinguish pattern variable qualifiers ("not supported") from
  FROM-clause range variable qualifiers ("not allowed").

  Variables appearing only in DEFINE (not in PATTERN) are also
  collected into `p_rpr_pattern_vars` so they get the "not supported"
  message, but the RPR_VARID_MAX count check still applies only to
  PATTERN variables.

[1]
https://www.postgresql.org/message-id/20260227.225456.33226875991025537.ishii@postgresql.org
[2]
https://www.postgresql.org/message-id/20260227.145539.1921177948671828231.ishii@postgresql.org

Attachment:
  nocfbot-0011-define-cost.txt
  nocfbot-0012-qualified-refs.txt

Best regards,
Henson

  242. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-02T00:46:32Z

    Hi Henson,

> Hi Tatsuo,
> 
> Attached are two more incremental patches (nocfbot-0011, nocfbot-0012)
> on top of v43, continuing the nocfbot-0001..0010 series.
> 
> nocfbot-0011: Add RPR DEFINE expression cost to WindowAgg cost estimation
> 
>   Revised version of your cost estimation patch [1].  Fixes the
>   `(char *)` cast and moves DEFINE cost outside the windowFuncs loop.
> 
> nocfbot-0012: Reject qualified column references in RPR DEFINE clause
> 
>   Revised version of your qualified column reference patch [2].
>   Exposes `patternVarNames` via `ParseState.p_rpr_pattern_vars` to
>   distinguish pattern variable qualifiers ("not supported") from
>   FROM-clause range variable qualifiers ("not allowed").
> 
>   Variables appearing only in DEFINE (not in PATTERN) are also
>   collected into `p_rpr_pattern_vars` so they get the "not supported"
>   message, but the RPR_VARID_MAX count check still applies only to
>   PATTERN variables.
> 
> [1]
> https://www.postgresql.org/message-id/20260227.225456.33226875991025537.ishii@postgresql.org
> [2]
> https://www.postgresql.org/message-id/20260227.145539.1921177948671828231.ishii@postgresql.org
> 
> Attachment:
>   nocfbot-0011-define-cost.txt
>   nocfbot-0012-qualified-refs.txt

Thanks. I will try them out. Once they look good, shall I release v44
patch sets? Or do you have a plan to add more patches?

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  243. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-03-02T01:32:46Z

    Hi Tatsuo,

> Thanks. I will try them out. Once they look good, shall I release v44
> patch sets? Or do you have a plan to add more patches?

The NFA executor code looks stable to me at this point. I do have some
additional optimization ideas around absorption, but I think it's better
new features should be added more carefully, or deferred until after the
commit.

Please go ahead and release v44. Having a fresh patch set will also
make it easier to proceed with the next round of test modifications
and improvements.

Best regards,
Henson

  244. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-02T05:18:23Z

    >> Thanks. I will try them out. Once they look good, shall I release v44
>> patch sets? Or do you have a plan to add more patches?
> 
> The NFA executor code looks stable to me at this point. I do have some
> additional optimization ideas around absorption, but I think it's better
> new features should be added more carefully, or deferred until after the
> commit.
> 
> Please go ahead and release v44. Having a fresh patch set will also
> make it easier to proceed with the next round of test modifications
> and improvements.

Your patches look good. Attached are the v44 pathes, with minor fixes
to docs (mention that reluctant quantifiers are supported, exclusion
and empty PATTERN are not supported). Also makes clear the range of n
and m in {n, m} and other regular expression.

---- Major differences from v43 patches ----

- Implement reluctant quantifiers

- Disallow using PREV()/NEXT() in other than the DEFINE caluse

- Reject qualied column references in the DEFINE clause

- Fix explain test failures on 32bit platforms

- Fix and cleanup the NFA engine

- Fix docs that reluctant quantifiers are supported, exclusion and
  empty PATTERN are not supported.

- Some code refactorings

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  245. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-03-03T11:32:51Z

    Hi,Tatsuo

While reviewing the RPR test cases, I noticed that a subquery filter
on RPR window function results silently returns wrong results.

For example, given this query:

  SELECT * FROM (
      SELECT id, val, COUNT(*) OVER w as cnt
      FROM rpr_copy
      WINDOW w AS (
          ORDER BY id
          ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
          PATTERN (A B?)
          DEFINE A AS val > 10, B AS val > 20
      )
  ) sub
  WHERE cnt > 0
  ORDER BY id;

This should return 2 rows, but returns 0 rows instead.

The EXPLAIN plan shows that "cnt > 0" is pushed down into the
WindowAgg node as a Run Condition:

  WindowAgg
    Run Condition: (count(*) OVER w > 0)    <-- pushed down
    ->  Sort
          ->  Seq Scan on rpr_copy

I will investigate the cause and work on a fix.

Best regards,
Henson

  246. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-04T06:38:22Z

    Hi Henson,

> Hi,Tatsuo
> 
> While reviewing the RPR test cases, I noticed that a subquery filter
> on RPR window function results silently returns wrong results.
> 
> For example, given this query:
> 
>   SELECT * FROM (
>       SELECT id, val, COUNT(*) OVER w as cnt
>       FROM rpr_copy
>       WINDOW w AS (
>           ORDER BY id
>           ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>           PATTERN (A B?)
>           DEFINE A AS val > 10, B AS val > 20
>       )
>   ) sub
>   WHERE cnt > 0
>   ORDER BY id;
> 
> This should return 2 rows, but returns 0 rows instead.

Thanks for the report!

> The EXPLAIN plan shows that "cnt > 0" is pushed down into the
> WindowAgg node as a Run Condition:
> 
>   WindowAgg
>     Run Condition: (count(*) OVER w > 0)    <-- pushed down
>     ->  Sort
>           ->  Seq Scan on rpr_copy
> 
> I will investigate the cause and work on a fix.

I think there are two ways to solve the issue:

1) Fix the executor
2) Fix the planner

I tried #1 but I don't know how to fix it. The run condition is
already pushed in the window function. Existing code forces to stop
the aggregate evaluation if the run condition is not
satisfied. Without RPR, the optimization is correct because once the
run condition is not satisfied, there's no chance that the subsequence
rows satisfy the condition. But RPR is used, a partition/frame is
divided into multiple reduced frames and each should be evaluated to
the end of the partition/frame.

So, I tried #2 so that the planner does not push down the run
condition.

Attached is the patch.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  247. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-04T07:24:50Z

    Hi Henson,

>> The NFA executor code looks stable to me at this point. I do have some
>> additional optimization ideas around absorption, but I think it's better
>> new features should be added more carefully, or deferred until after the
>> commit.
>> 
>> Please go ahead and release v44. Having a fresh patch set will also
>> make it easier to proceed with the next round of test modifications
>> and improvements.

CFBot complains v44-0007 patch corrupted.
https://cfbot.cputube.org/patch_4460.log

In my local environment, the patch size was:
-rw-rw-r-- 1 t-ishii t-ishii 966311  3月  4 16:10 /home/t-ishii/v44-0007-Row-pattern-recognition-patch-tests.patch
While the file downloaded from PostgreSQL mailing list archive was:
-rw-rw-r-- 1 t-ishii t-ishii 250064  3月  4 16:15 v44-0007-Row-pattern-recognition-patch-tests.patch

So it seems the file was truncated somwhere between me and PostgreSQL
ML archiving.

Can you check the size of the file attached to your mail?

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  248. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-03-04T07:29:35Z

    Hi Tatsuo,

CFBot complains v44-0007 patch corrupted.
> https://cfbot.cputube.org/patch_4460.log


I downloaded v44-0007 from the mailing list archive and applied it
successfully in my local environment. Everything works fine here.


> So it seems the file was truncated somwhere between me and PostgreSQL
> ML archiving.
>

It seems CFBot does not retry once it fails, so the initial truncation
error has become permanent.


> Can you check the size of the file attached to your mail?


I didn't check the raw attachment size, but the patch applied cleanly
without any issues.

To work around this, how about splitting the test patch (0007) into
separate patches for .sql and .out files? This would reduce the
individual file sizes and help avoid truncation during archiving.

Best regards,
Henson

  249. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-04T08:12:03Z

    Hi Henson,

> I downloaded v44-0007 from the mailing list archive and applied it
> successfully in my local environment. Everything works fine here.

Oh, ok.
I tried to download the file again, and now the file size increased!
It seems the ML downloading feature is a little bit instable?

> I didn't check the raw attachment size, but the patch applied cleanly
> without any issues.
> 
> To work around this, how about splitting the test patch (0007) into
> separate patches for .sql and .out files? This would reduce the
> individual file sizes and help avoid truncation during archiving.

Yeah, let's give it a try.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  250. Re: Row pattern recognition

    SungJun Jang <sjjang112233@gmail.com> — 2026-03-05T04:16:21Z

    Hi hackers

I converted PostgreSQL RPR regression test queries to Oracle
MATCH_RECOGNIZE syntax and executed them on both systems to perform
cross-validation.

The tests were based on the following PostgreSQL regression test files:

  rpr_base.sql
  rpr_nfa.sql


PostgreSQL bug: zero-min reluctant quantifier

During cross-validation a PostgreSQL bug was discovered involving
reluctant quantifiers whose minimum repetition is 0.

Example pattern:

  PATTERN (A*?)
  DEFINE A AS val > 0

Result comparison:

pattern   PostgreSQL (cnt)   Oracle (cnt)
A*?       1,1,1              0,0,0
A??       1,1,1              0,0,0
A+?       1,1,1              1,1,1

For reluctant quantifiers with min=0 (such as *? and ??), PostgreSQL
always consumes at least one row, while Oracle allows a zero-length
match. When min>=1 (e.g., A+?), both systems behave the same.

This behavior was consistently observed across the converted tests.


Design difference: unused DEFINE variables

Example:

  PATTERN (A+)
  DEFINE A AS id > 0, B AS id > 5

PostgreSQL executes the query successfully and ignores the unused
variable B.

Oracle raises:

  ORA-62503: illegal variable definition


Oracle limitations observed


Bounded quantifier limit

A{200}  -> works
A{201}  -> ORA-62518

Oracle appears to limit the upper bound of bounded quantifiers to 200,
while PostgreSQL does not impose this restriction.


Nested nullable quantifiers

Examples:

  (A*)*
  (A*)+
  (((A)*)*)*

  (A?|B){1,2}
  ((A?){2,3}){2,3}
  (A?){n,m}
  (A? B?){2,3}

Oracle raises:

  ORA-62513

when a nullable subpattern is wrapped by an outer quantifier, while
PostgreSQL executes these patterns successfully.


These results come from running the converted PostgreSQL RPR regression
tests on Oracle for comparison.

Best regards,
SugJun

  251. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-03-05T04:59:41Z

    Hi Tatsuo,

PostgreSQL bug: zero-min reluctant quantifier
>
> During cross-validation a PostgreSQL bug was discovered involving
> reluctant quantifiers whose minimum repetition is 0.
>
> Example pattern:
>
>   PATTERN (A*?)
>   DEFINE A AS val > 0
>
> Result comparison:
>
> pattern   PostgreSQL (cnt)   Oracle (cnt)
> A*?       1,1,1              0,0,0
> A??       1,1,1              0,0,0
> A+?       1,1,1              1,1,1
>
> For reluctant quantifiers with min=0 (such as *? and ??), PostgreSQL
> always consumes at least one row, while Oracle allows a zero-length
> match. When min>=1 (e.g., A+?), both systems behave the same.
>

This is indeed a bug. Thanks SugJun for finding it. I'll fix this in
the next patch.

Design difference: unused DEFINE variables
>
> Example:
>
>   PATTERN (A+)
>   DEFINE A AS id > 0, B AS id > 5
>
> PostgreSQL executes the query successfully and ignores the unused
> variable B.
>
> Oracle raises:
>
>   ORA-62503: illegal variable definition
>

Currently PostgreSQL silently removes unused DEFINE variables during
optimization. Do you think we should raise an error instead, as Oracle
does?


> Oracle limitations observed
>
>
> Bounded quantifier limit
>
> A{200}  -> works
> A{201}  -> ORA-62518
>
> Oracle appears to limit the upper bound of bounded quantifiers to 200,
> while PostgreSQL does not impose this restriction.
>

I don't think we need to impose an artificial limit like Oracle's 200.
What do you think?


> Nested nullable quantifiers
>
> Examples:
>
>   (A*)*
>   (A*)+
>   (((A)*)*)*
>
>   (A?|B){1,2}
>   ((A?){2,3}){2,3}
>   (A?){n,m}
>   (A? B?){2,3}
>
> Oracle raises:
>
>   ORA-62513
>
> when a nullable subpattern is wrapped by an outer quantifier, while
> PostgreSQL executes these patterns successfully.
>

This seems like an Oracle limitation rather than a standard requirement.

Best regards,
Henson

>

  252. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-05T05:20:49Z

    Hi Henson,

> Hi Tatsuo,
> 
> PostgreSQL bug: zero-min reluctant quantifier
>>
>> During cross-validation a PostgreSQL bug was discovered involving
>> reluctant quantifiers whose minimum repetition is 0.
>>
>> Example pattern:
>>
>>   PATTERN (A*?)
>>   DEFINE A AS val > 0
>>
>> Result comparison:
>>
>> pattern   PostgreSQL (cnt)   Oracle (cnt)
>> A*?       1,1,1              0,0,0
>> A??       1,1,1              0,0,0
>> A+?       1,1,1              1,1,1
>>
>> For reluctant quantifiers with min=0 (such as *? and ??), PostgreSQL
>> always consumes at least one row, while Oracle allows a zero-length
>> match. When min>=1 (e.g., A+?), both systems behave the same.
>>
> 
> This is indeed a bug. Thanks SugJun for finding it. I'll fix this in
> the next patch.

Thanks in advance.

> Design difference: unused DEFINE variables
>>
>> Example:
>>
>>   PATTERN (A+)
>>   DEFINE A AS id > 0, B AS id > 5
>>
>> PostgreSQL executes the query successfully and ignores the unused
>> variable B.
>>
>> Oracle raises:
>>
>>   ORA-62503: illegal variable definition
>>
> 
> Currently PostgreSQL silently removes unused DEFINE variables during
> optimization. Do you think we should raise an error instead, as Oracle
> does?

No, I don't think so. I think the standard does not say anything if a
pattern variable defined in DEFINE clause is not used in PATTERN
clause. So the expected behavior would be implementation dependent. I
think just ignoring the variable is fine.

>> Oracle limitations observed
>>
>>
>> Bounded quantifier limit
>>
>> A{200}  -> works
>> A{201}  -> ORA-62518
>>
>> Oracle appears to limit the upper bound of bounded quantifiers to 200,
>> while PostgreSQL does not impose this restriction.
>>
> 
> I don't think we need to impose an artificial limit like Oracle's 200.
> What do you think?

Agreed. We do not need to follow Oracle here.

>> Nested nullable quantifiers
>>
>> Examples:
>>
>>   (A*)*
>>   (A*)+
>>   (((A)*)*)*
>>
>>   (A?|B){1,2}
>>   ((A?){2,3}){2,3}
>>   (A?){n,m}
>>   (A? B?){2,3}
>>
>> Oracle raises:
>>
>>   ORA-62513
>>
>> when a nullable subpattern is wrapped by an outer quantifier, while
>> PostgreSQL executes these patterns successfully.
>>
> 
> This seems like an Oracle limitation rather than a standard requirement.

Agreed. For example, the standard explicitly stats A(*)* is permitted
(ISO/IEC 19075-5, 4.1.4.1 Introduction to the PATTEREN syntax).

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  253. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-03-06T04:38:30Z

    Hi Tatsuo,

I reviewed the planner's Window clause optimizations and found
additional issues with RPR windows.  Thanks to SungJun's Oracle
cross-validation work, all converted regression tests now pass
on both systems.  With these fixes, I'd like to move on to
non-functional quality testing next.

Here are eight incremental patches on top of v44.


nocfbot-0001: Fix elog message to use lowercase per PostgreSQL convention

  Minor style fix: lowercase the elog error message in
  nodeWindowAgg.c to follow PostgreSQL coding convention.


nocfbot-0002: Fix RPR reluctant quantifier flag lost during VIEW
serialization

  The reluctant flag on quantifiers was lost when a VIEW containing
  RPR was serialized and restored via ruleutils.c.  This patch fixes
  gram.y, rpr.c, ruleutils.c, and parsenodes.h so the flag survives
  the round-trip.


nocfbot-0003: Expand RPR test coverage and improve test comments

  Reorganizes test cases across rpr.sql, rpr_base.sql, rpr_explain.sql,
  and rpr_nfa.sql.  Moves base functionality tests from rpr.sql to
  rpr_base.sql and NFA-specific tests to rpr_nfa.sql.  Adds better
  section comments throughout.  Duplicate tests in rpr.sql were
  removed.  No reduction in test coverage.


nocfbot-0004: Keep RPR test objects for pg_upgrade/pg_dump testing

  Adjusts rpr_base.sql and rpr_explain.sql so that test tables and
  views are not dropped at the end of the test.  This allows
  pg_upgrade and pg_dump regression testing to cover RPR objects.


nocfbot-0005: Disable run condition pushdown for RPR windows

  Revised version of your run condition fix [1].  Existing test
  expected output updated accordingly.


nocfbot-0006: Disable frame optimization for RPR windows

  Prevents the planner from optimizing the window frame for RPR
  windows.  The frame clause in RPR has different semantics (it
  bounds the pattern search space), so standard frame optimizations
  must be disabled.  Adds EXPLAIN tests to verify.  Please review
  this one -- it's a newly discovered issue.


nocfbot-0007: Add stock scenario tests for RPR pattern matching

  Adds stock trading scenario tests using realistic synthetic data
  (stock.data with 1632 rows).  Tests V-shape recovery, W-shape,
  consecutive rises, and other common pattern matching use cases.


nocfbot-0008: Fix zero-min reluctant quantifier to produce zero-length match

  Fixes the bug reported by SungJun [2]: reluctant quantifiers with
  min=0 (A*?, A??) were incorrectly consuming at least one row instead
  of producing a zero-length match.  The NFA can now internally
  distinguish between a zero-length match and no match, so it will
  be ready when additional infrastructure (e.g. MATCH_NUMBER) is
  added.  Now matches Oracle behavior.


[1]
https://www.postgresql.org/message-id/20260304.153822.445473532741409674.ishii@postgresql.org
[2]
https://www.postgresql.org/message-id/CAE+cgNiUbKeH1A0PoxV2QjpsoxJLe+pJcGz_gdxwOwu_9zqchw@mail.gmail.com


Best regards,
Henson

  254. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-06T06:38:37Z

    Hi Henson,

> Hi Tatsuo,
> 
> I reviewed the planner's Window clause optimizations and found
> additional issues with RPR windows.  Thanks to SungJun's Oracle
> cross-validation work, all converted regression tests now pass
> on both systems.  With these fixes, I'd like to move on to
> non-functional quality testing next.

Sounds good.

> Here are eight incremental patches on top of v44.
> 
> 
> nocfbot-0001: Fix elog message to use lowercase per PostgreSQL convention
> 
>   Minor style fix: lowercase the elog error message in
>   nodeWindowAgg.c to follow PostgreSQL coding convention.

Thanks.

> nocfbot-0002: Fix RPR reluctant quantifier flag lost during VIEW
> serialization
> 
>   The reluctant flag on quantifiers was lost when a VIEW containing
>   RPR was serialized and restored via ruleutils.c.  This patch fixes
>   gram.y, rpr.c, ruleutils.c, and parsenodes.h so the flag survives
>   the round-trip.

Great.

> nocfbot-0003: Expand RPR test coverage and improve test comments
> 
>   Reorganizes test cases across rpr.sql, rpr_base.sql, rpr_explain.sql,
>   and rpr_nfa.sql.  Moves base functionality tests from rpr.sql to
>   rpr_base.sql and NFA-specific tests to rpr_nfa.sql.  Adds better
>   section comments throughout.  Duplicate tests in rpr.sql were
>   removed.  No reduction in test coverage.

Sounds good.

> nocfbot-0004: Keep RPR test objects for pg_upgrade/pg_dump testing
> 
>   Adjusts rpr_base.sql and rpr_explain.sql so that test tables and
>   views are not dropped at the end of the test.  This allows
>   pg_upgrade and pg_dump regression testing to cover RPR objects.

Great.

> nocfbot-0005: Disable run condition pushdown for RPR windows
> 
>   Revised version of your run condition fix [1].  Existing test
>   expected output updated accordingly.

Thanks for including the patch.

> nocfbot-0006: Disable frame optimization for RPR windows
> 
>   Prevents the planner from optimizing the window frame for RPR
>   windows.  The frame clause in RPR has different semantics (it
>   bounds the pattern search space), so standard frame optimizations
>   must be disabled.  Adds EXPLAIN tests to verify.  Please review
>   this one -- it's a newly discovered issue.

Oh, this is a very important finding! I will definitely look into the
patch.

> nocfbot-0007: Add stock scenario tests for RPR pattern matching
> 
>   Adds stock trading scenario tests using realistic synthetic data
>   (stock.data with 1632 rows).  Tests V-shape recovery, W-shape,
>   consecutive rises, and other common pattern matching use cases.

Excellent.

> nocfbot-0008: Fix zero-min reluctant quantifier to produce zero-length match
> 
>   Fixes the bug reported by SungJun [2]: reluctant quantifiers with
>   min=0 (A*?, A??) were incorrectly consuming at least one row instead
>   of producing a zero-length match.  The NFA can now internally
>   distinguish between a zero-length match and no match, so it will
>   be ready when additional infrastructure (e.g. MATCH_NUMBER) is
>   added.  Now matches Oracle behavior.

Does "a zero-length match" mean "an empty match"?

BTW, currently we place all nfa_* functions at the bottom of
nodeWindowAgg.c.  However nodeWindowAgg.c in master branch places "API
exposed to window functions" at the bottom of the file. Do you think
we should follow the way?

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  255. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-03-06T23:18:27Z

    Hi, Tatsuo

Does "a zero-length match" mean "an empty match"?
>

Yes, they refer to the same thing.  "Zero-length match" is the more
common term in general regex implementations (PCRE2, Perl, Python,
Java, etc.[1]), but the RPR standard (ISO/IEC 19075-5, Section 4.12.2)
uses "empty match" exclusively.

[1] https://www.regular-expressions.info/zerolength.html


BTW, currently we place all nfa_* functions at the bottom of
> nodeWindowAgg.c.  However nodeWindowAgg.c in master branch places "API
> exposed to window functions" at the bottom of the file. Do you think
> we should follow the way?


Yes, we should follow master's convention.  I see three options:

  (a) Reorder within nodeWindowAgg.c: move the nfa_* functions up and
      keep the "API exposed to window functions" section at the bottom,
      matching master's layout.

  (b) Separate file under src/backend/executor/, keeping it close to
      nodeWindowAgg.c while making the boundary explicit.

  (c) A dedicated src/backend/rpr/ directory modeled on
      src/backend/regex/, giving the NFA engine its own namespace.
      This could also be an opportunity to consolidate the existing
      src/backend/optimizer/plan/rpr.c into the same directory.

For now (a) is the safest change.  Longer term, (b) or (c) would make
more sense -- especially when we extend to MATCH_RECOGNIZE (R010),
where the NFA engine will need to be shared across both code paths.
Either way, the NFA engine can be exposed via a header so that R010
can share it without further restructuring.

Since the NFA algorithm is not familiar territory for most DBMS
developers, it would also be worth preserving the detailed algorithm
description posted earlier in this thread -- either as structured
comments or as a dedicated README alongside the code.

What do you think?  Should we start with (a) now and revisit the
broader restructuring approaches -- (b) or (c) -- later, or would you
prefer to discuss them first?  Either of those would also resolve the
file layout convention issue naturally, since new files would follow
proper conventions from the start.


One more thing: there are no ECPG example programs or regression tests
for RPR yet.  I'd like to propose adding them.  Shall I draft an
initial set, or would you prefer to coordinate with the ECPG
maintainers first?


Best regards,
Henson

  256. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-07T03:01:51Z

    Hi Henson,

> Hi, Tatsuo
> 
> Does "a zero-length match" mean "an empty match"?
>>
> 
> Yes, they refer to the same thing.  "Zero-length match" is the more
> common term in general regex implementations (PCRE2, Perl, Python,
> Java, etc.[1]), but the RPR standard (ISO/IEC 19075-5, Section 4.12.2)
> uses "empty match" exclusively.
> 
> [1] https://www.regular-expressions.info/zerolength.html

I found Trino uses "empty match" too [2]. So for SQL users, I guess
"empty match" is more familiar wording.

> Yes, we should follow master's convention.  I see three options:
> 
>   (a) Reorder within nodeWindowAgg.c: move the nfa_* functions up and
>       keep the "API exposed to window functions" section at the bottom,
>       matching master's layout.
> 
>   (b) Separate file under src/backend/executor/, keeping it close to
>       nodeWindowAgg.c while making the boundary explicit.
> 
>   (c) A dedicated src/backend/rpr/ directory modeled on
>       src/backend/regex/, giving the NFA engine its own namespace.
>       This could also be an opportunity to consolidate the existing
>       src/backend/optimizer/plan/rpr.c into the same directory.
> 
> For now (a) is the safest change.  Longer term, (b) or (c) would make
> more sense -- especially when we extend to MATCH_RECOGNIZE (R010),
> where the NFA engine will need to be shared across both code paths.
> Either way, the NFA engine can be exposed via a header so that R010
> can share it without further restructuring.
> 
> Since the NFA algorithm is not familiar territory for most DBMS
> developers, it would also be worth preserving the detailed algorithm
> description posted earlier in this thread -- either as structured
> comments or as a dedicated README alongside the code.
> 
> What do you think?  Should we start with (a) now and revisit the
> broader restructuring approaches -- (b) or (c) -- later, or would you
> prefer to discuss them first?  Either of those would also resolve the
> file layout convention issue naturally, since new files would follow
> proper conventions from the start.

I prefer (a) or (b) for now, at least for the first commit. The reason
is, current nfa functions take a WindowAggState argument. If we prefer
(c), I think we need to change some of (or most of) nfa functions so
that they do not take the WindowAggState argument. What do you think?

> One more thing: there are no ECPG example programs or regression tests
> for RPR yet.  I'd like to propose adding them.  Shall I draft an
> initial set, or would you prefer to coordinate with the ECPG
> maintainers first?

I am not familiar with ECPG. Do you know if ECPG has Window clause
tests? If ECPG does not have any Window clause tests, is it worth to
add RPR tests to ECPG?

[2] https://trino.io/docs/current/sql/match-recognize.html#evaluating-expressions-in-empty-matches-and-unmatched-rows

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  257. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-03-07T16:02:41Z

    Hi Tatsuo,

I found Trino uses "empty match" too [2]. So for SQL users, I guess
> "empty match" is more familiar wording.
>

Agreed.  Since RPR is a feature defined by the SQL standard, using the
standard's own terminology is the right choice: it lets users
cross-reference the specification directly.  That said, "zero-length
match" is widely used in the regex literature (PCRE2, Perl, Python,
Java, etc.), so developers researching the topic should treat the two
terms as synonymous.  In the SQL context, "empty match" is the preferred
term.



> >   (a) Reorder within nodeWindowAgg.c: move the nfa_* functions up and
> >       keep the "API exposed to window functions" section at the bottom,
> >       matching master's layout.
> >
> >   (b) Separate file under src/backend/executor/, keeping it close to
> >       nodeWindowAgg.c while making the boundary explicit.
> >
> >   (c) A dedicated src/backend/rpr/ directory modeled on
> >       src/backend/regex/, giving the NFA engine its own namespace.
> >       This could also be an opportunity to consolidate the existing
> >       src/backend/optimizer/plan/rpr.c into the same directory.
>
> I prefer (a) or (b) for now, at least for the first commit. The reason
> is, current nfa functions take a WindowAggState argument. If we prefer
> (c), I think we need to change some of (or most of) nfa functions so
> that they do not take the WindowAggState argument. What do you think?
>

That is a valid concern.  (c) will be necessary in the long run, but
MATCH_RECOGNIZE (R010) is not yet on the horizon, and restructuring the
NFA layer now -- before that requirement is concrete -- carries a real
risk of introducing defects during the refactoring.  The risks outweigh the
benefits at this stage.

Instead, I'll try (b): a separate file under src/backend/executor/
can retain the WindowAggState argument while making the NFA boundary
explicit, and the change is still reviewable.  If (b) turns out to be
harder than expected or surfaces unexpected issues, I'll fall back to
(a).

If (b) works out, I'll add a structured header comment to the new file
to preserve the algorithm description for future reviewers.


I am not familiar with ECPG. Do you know if ECPG has Window clause
> tests? If ECPG does not have any Window clause tests, is it worth to
> add RPR tests to ECPG?
>

I checked src/interfaces/ecpg/test/ and confirmed there are no Window
clause tests there at all.  If ECPG has no Window clause coverage, I
see no reason to add RPR tests there at this stage.  I think it would be
better to leave ECPG tests out of this patch set for now.


Best regards,
Henson

  258. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-07T23:47:22Z

    Hi Henson,

>> nocfbot-0006: Disable frame optimization for RPR windows
>> 
>>   Prevents the planner from optimizing the window frame for RPR
>>   windows.  The frame clause in RPR has different semantics (it
>>   bounds the pattern search space), so standard frame optimizations
>>   must be disabled.  Adds EXPLAIN tests to verify.  Please review
>>   this one -- it's a newly discovered issue.
> 
> Oh, this is a very important finding! I will definitely look into the
> patch.

As I promised, I looked into nocfbot-0006 and it is good to me.

While at it, I looked around optimize_window_clauses and encountered
this:
				/*
				 * Perform the same duplicate check that is done in
				 * transformWindowFuncCall.
				 */
				if (equal(wc->partitionClause, existing_wc->partitionClause) &&
					equal(wc->orderClause, existing_wc->orderClause) &&
					wc->frameOptions == existing_wc->frameOptions &&
					equal(wc->startOffset, existing_wc->startOffset) &&
					equal(wc->endOffset, existing_wc->endOffset))

It looks like it missed the check for DEINFE and PATTERN clause but
with your patch, if RPR is defined we do not come here anyway. So, I
looked in transformWindowFuncCall (parse_agg.c) as it mentioned in the
comment above and I found this:

			/*
			 * Also see similar de-duplication code in optimize_window_clauses
			 */
			if (equal(refwin->partitionClause, windef->partitionClause) &&
				equal(refwin->orderClause, windef->orderClause) &&
				refwin->frameOptions == windef->frameOptions &&
				equal(refwin->startOffset, windef->startOffset) &&
				equal(refwin->endOffset, windef->endOffset))

This basically combines a window definition without window name into
an existing window definition if they shar the identical window
definition. For example:

CREATE TEMP TABLE rpr_copy (id INT, val INT);
INSERT INTO rpr_copy VALUES (1, 10), (2, 20), (3, 30), (4, 40);
SELECT id, val, first_value(id) OVER (
       ORDER BY id
       ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
), first_value(id) OVER w1
FROM rpr_copy
WINDOW w1 AS (
       ORDER BY id
       ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
       PATTERN (A+)
       DEFINE A AS val > 10
);

The two window definitions are regarded as identical and produces
wrong result.

 id | val | first_value | first_value 
----+-----+-------------+-------------
  1 |  10 |             |            
  2 |  20 |           2 |           2
  3 |  30 |             |            
  4 |  40 |             |            
(4 rows)

Of course proper result is:

 id | val | first_value | first_value 
----+-----+-------------+-------------
  1 |  10 |           1 |            
  2 |  20 |           2 |           2
  3 |  30 |           3 |            
  4 |  40 |           4 |            
(4 rows)

Explain result:

                                   QUERY PLAN                                   
--------------------------------------------------------------------------------
 WindowAgg  (cost=209.33..215.01 rows=2260 width=17)
   Window: w1 AS (ORDER BY id ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
   Pattern: a+"
   ->  Sort  (cost=158.51..164.16 rows=2260 width=8)
         Sort Key: id
         ->  Seq Scan on rpr_copy  (cost=0.00..32.60 rows=2260 width=8)
(6 rows)

The two window definitions are unified into w1, which has the RPR
part, which is wrong.  To fix this, we need to check the RPR part
while unifying the window definitions. Attached patch does it (on top
of v44).

Question is, should we fix optimize_window_clause as well? As I
mentioned earlier, with Henson's fix it does not produce any bug at
this moment. However, we may want to fix for code consistency.  Not
fixing it will break the comment.

Thoughts?

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  259. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-03-08T14:24:55Z

    Hi Tatsuo,

> Question is, should we fix optimize_window_clause as well?

Yes, we should. The comment in optimize_window_clauses explicitly
states "Perform the same duplicate check that is done in
transformWindowFuncCall", so the two checks must remain in sync.
Even if omitting the RPR fields produces no visible bug today (because
the early return guard in nocfbot-0006 already skips RPR windows), the
code would be misleading and fragile: a future change could remove
that guard and silently reintroduce the bug.

Adding the RPR fields explicitly also serves as defensive coding and
makes the intent clear to anyone reading or modifying the code later.

When incorporating nocfbot-0006 as patch 06/12, this fix was included:

    wc->rpSkipTo == existing_wc->rpSkipTo &&
    wc->initial == existing_wc->initial &&
    equal(wc->defineClause, existing_wc->defineClause) &&
    equal(wc->rpPattern, existing_wc->rpPattern)

Patch 06/12 now covers both the frame optimization guard and the
optimize_window_clauses deduplication fix, with EXPLAIN-based regression
tests added to verify the behavior.

> The two window definitions are regarded as identical and produces
> wrong result.

Thank you for finding this. Your patch has been incorporated as patch
09/12 ("Fix window deduplication for RPR windows at parse time"),
modified to add regression test cases covering the scenario. The fix
adds the missing RPR check in transformWindowFuncCall:

    equal(refwin->rpCommonSyntax, windef->rpCommonSyntax)

This prevents incorrect merging of an RPR window with a non-RPR window
that has identical frame options.

The previous submission (nocfbot-0001..0008) has been extended to
12 patches. Changes since the last submission:

  06/12  Disable frame optimization for RPR windows  [updated]
         optimize_window_clauses deduplication fix added, as discussed.

  09/12  Fix window deduplication for RPR windows at parse time  [new]
         Incorporates your fix with test cases added.

  10/12  Walk DEFINE clause in window tree traversal  [new]
         A newly discovered issue: nodeFuncs.c was not visiting the
         DEFINE clause in expression_tree_walker, query_tree_walker,
         and their mutator counterparts. The demonstrated case is SQL
         function inlining: a SQL function with a parameter used in
         DEFINE (e.g., DEFINE A AS v > $1) would fail to substitute
         the actual argument, producing wrong results.

  11/12  Use SQL standard term "empty match" in RPR comments  [new]
         Align NFA comment terminology with the SQL standard: replace
         implementation-specific terms "zero-length match" and
         "zero-consumption" with the standard term "empty match".

  12/12  Extract RPR NFA engine into execRPR.c  [new]
         Following the earlier discussion about file layout, this
         implements option (b): move the NFA engine to a dedicated
         src/backend/executor/execRPR.c, keeping it close to
         nodeWindowAgg.c while making the boundary explicit. Functions
         still take WindowAggState as an argument (no signature change).
         The public interface is declared in execRPR.h using the ExecRPR*
         naming convention, consistent with other executor headers.
         The NFA algorithm guide shared earlier in this thread has also
         been incorporated as structured comments at the top of the file,
         as suggested.

The full updated series (12 patches on top of v44) is attached.

Regards,
Henson

  260. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-09T03:08:02Z

    Hi Henson,

>> Question is, should we fix optimize_window_clause as well?
> 
> Yes, we should. The comment in optimize_window_clauses explicitly
> states "Perform the same duplicate check that is done in
> transformWindowFuncCall", so the two checks must remain in sync.
> Even if omitting the RPR fields produces no visible bug today (because
> the early return guard in nocfbot-0006 already skips RPR windows), the
> code would be misleading and fragile: a future change could remove
> that guard and silently reintroduce the bug.
> 
> Adding the RPR fields explicitly also serves as defensive coding and
> makes the intent clear to anyone reading or modifying the code later.

Agreed.

> When incorporating nocfbot-0006 as patch 06/12, this fix was included:
> 
>     wc->rpSkipTo == existing_wc->rpSkipTo &&
>     wc->initial == existing_wc->initial &&
>     equal(wc->defineClause, existing_wc->defineClause) &&
>     equal(wc->rpPattern, existing_wc->rpPattern)
> 
> Patch 06/12 now covers both the frame optimization guard and the
> optimize_window_clauses deduplication fix, with EXPLAIN-based regression
> tests added to verify the behavior.

Good.

> Thank you for finding this. Your patch has been incorporated as patch
> 09/12 ("Fix window deduplication for RPR windows at parse time"),
> modified to add regression test cases covering the scenario. The fix
> adds the missing RPR check in transformWindowFuncCall:
> 
>     equal(refwin->rpCommonSyntax, windef->rpCommonSyntax)
> 
> This prevents incorrect merging of an RPR window with a non-RPR window
> that has identical frame options.

Good.

> The previous submission (nocfbot-0001..0008) has been extended to
> 12 patches. Changes since the last submission:
> 
>   06/12  Disable frame optimization for RPR windows  [updated]
>          optimize_window_clauses deduplication fix added, as discussed.
>
>   09/12  Fix window deduplication for RPR windows at parse time  [new]
>          Incorporates your fix with test cases added.

Thanks for adding test cases.

>   10/12  Walk DEFINE clause in window tree traversal  [new]
>          A newly discovered issue: nodeFuncs.c was not visiting the
>          DEFINE clause in expression_tree_walker, query_tree_walker,
>          and their mutator counterparts. The demonstrated case is SQL
>          function inlining: a SQL function with a parameter used in
>          DEFINE (e.g., DEFINE A AS v > $1) would fail to substitute
>          the actual argument, producing wrong results.

Excellnt findings!  BTW, I realized that we cannot use $1 of function
in PATTERN clause like: A{$1}.

ERROR:  42601: syntax error at or near "$1"
LINE 10:         PATTERN (A{$1})
                            ^
LOCATION:  scanner_yyerror, scan.l:1211

Should we document somewhere?

>   11/12  Use SQL standard term "empty match" in RPR comments  [new]
>          Align NFA comment terminology with the SQL standard: replace
>          implementation-specific terms "zero-length match" and
>          "zero-consumption" with the standard term "empty match".

Ok.

>   12/12  Extract RPR NFA engine into execRPR.c  [new]
>          Following the earlier discussion about file layout, this
>          implements option (b): move the NFA engine to a dedicated
>          src/backend/executor/execRPR.c, keeping it close to
>          nodeWindowAgg.c while making the boundary explicit. Functions
>          still take WindowAggState as an argument (no signature change).
>          The public interface is declared in execRPR.h using the ExecRPR*
>          naming convention, consistent with other executor headers.
>          The NFA algorithm guide shared earlier in this thread has also
>          been incorporated as structured comments at the top of the file,
>          as suggested.

Excellent work! Now the exported NFA engine API is cleary separated
from NFA internal static fnctions, which greatly enhances the
readability.

> The full updated series (12 patches on top of v44) is attached.

Thank you! Patch applied cleanly and passed all the regression tests.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  261. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-03-09T04:02:02Z

    Hi Tatsuo,


> >   10/12  Walk DEFINE clause in window tree traversal  [new]
> >          A newly discovered issue: nodeFuncs.c was not visiting the
> >          DEFINE clause in expression_tree_walker, query_tree_walker,
> >          and their mutator counterparts. The demonstrated case is SQL
> >          function inlining: a SQL function with a parameter used in
> >          DEFINE (e.g., DEFINE A AS v > $1) would fail to substitute
> >          the actual argument, producing wrong results.
>
> Excellnt findings!  BTW, I realized that we cannot use $1 of function
> in PATTERN clause like: A{$1}.
>
> ERROR:  42601: syntax error at or near "$1"
> LINE 10:         PATTERN (A{$1})
>                             ^
> LOCATION:  scanner_yyerror, scan.l:1211
>
> Should we document somewhere?
>

The PATTERN quantifier {n} only accepts Iconst (integer literal) in the
grammar.  When a host variable or function parameter is used (e.g.,
A{$1}), the user gets a generic syntax error.

Oracle accepts broader syntax and validates later, producing an error
at a later stage rather than a syntax error at parse time.

PostgreSQL itself already has precedent for this pattern -- in fact,
within the same window clause, frame offset (ROWS/RANGE/GROUPS) accepts
a_expr in the grammar and then rejects variables in parse analysis via
transformFrameOffset() -> checkExprIsVarFree().

I'd lean against documenting this.  The SQL standard already defines
the quantifier bound as <unsigned integer literal>, so there is nothing
beyond the standard to call out, and documenting what is *not* allowed
tends to raise questions that wouldn't otherwise occur to users.

Rather, I think accepting a broader grammar and validating later would
be the more appropriate response, producing a descriptive error like:

  "argument of bounded quantifier must be an integer literal"

I can either include this in the current patch set or handle it as a
separate follow-up after the main series is committed.  What do you
think?

Regards,
Henson

  262. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-09T05:22:02Z

    Hi Henson,

>> Excellnt findings!  BTW, I realized that we cannot use $1 of function
>> in PATTERN clause like: A{$1}.
>>
>> ERROR:  42601: syntax error at or near "$1"
>> LINE 10:         PATTERN (A{$1})
>>                             ^
>> LOCATION:  scanner_yyerror, scan.l:1211
>>
>> Should we document somewhere?
>>
> 
> The PATTERN quantifier {n} only accepts Iconst (integer literal) in the
> grammar.  When a host variable or function parameter is used (e.g.,
> A{$1}), the user gets a generic syntax error.

Ok.

> Oracle accepts broader syntax and validates later, producing an error
> at a later stage rather than a syntax error at parse time.
> 
> PostgreSQL itself already has precedent for this pattern -- in fact,
> within the same window clause, frame offset (ROWS/RANGE/GROUPS) accepts
> a_expr in the grammar and then rejects variables in parse analysis via
> transformFrameOffset() -> checkExprIsVarFree().
> 
> I'd lean against documenting this.  The SQL standard already defines
> the quantifier bound as <unsigned integer literal>, so there is nothing
> beyond the standard to call out, and documenting what is *not* allowed
> tends to raise questions that wouldn't otherwise occur to users.
> 
> Rather, I think accepting a broader grammar and validating later would
> be the more appropriate response, producing a descriptive error like:
> 
>   "argument of bounded quantifier must be an integer literal"
> 
> I can either include this in the current patch set or handle it as a
> separate follow-up after the main series is committed.  What do you
> think?

I think handing it as a separate follow-up after the commit is enough
unless other developers complain.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  263. Re: Row pattern recognition

    SungJun Jang <sjjang112233@gmail.com> — 2026-03-11T04:37:59Z

    Hi hackers,

I ran a cross-validation test comparing PostgreSQL RPR (current patch) and
Trino MATCH_RECOGNIZE to verify both result correctness and performance
characteristics across different scales.

The dataset consists of two segments where rows are arranged sequentially
as:

A → B → C → D

Each segment contains a fixed distribution of categories A/B/C/D. Category
E does not exist in the dataset and is used to test match failure behavior.

Test scales ranged from 20,000 to 100,000 rows.

All tests were executed using:

AFTER MATCH SKIP PAST LAST ROW

Test Environment

Item        | PostgreSQL              | Trino
------------+-------------------------+-----------------------
Version     | 19devel (RPR patch)    | 471
Runtime     | Local                  | Local Docker container
Platform    | Linux x86_64           | Linux x86_64
RPR syntax  | WINDOW clause          | MATCH_RECOGNIZE

Dataset Structure

Each segment contains sequential categories:

A : ~1/3 of rows
B : ~1/3 of rows
C : ~1/3 of rows
D : 1 row

Example at 1x scale:

Segment size : 10,000 rows
Total rows : 20,000

Test Cases

Test 1
Pattern:
PATTERN (A+ B+ C+ D)

Expected result:
1 match per segment (2 total)

Reason:
Data is arranged exactly as A → B → C → D.

Test 2
Pattern:
PATTERN (A+ B+ C+ E)

Expected result:
0 rows (no match)

Reason:
Category E does not exist in the dataset.

Correctness Results

Across all scales both systems returned identical results.

Test 1 → 1 match per segment (2 total)
Test 2 → 0 rows

Performance Results

Scale | Total Rows | PG Test1 | PG Test2 | Trino Test1 | Trino Test2
------+------------+----------+----------+-------------+-------------
1x    |  20,000    | 19 ms    | 17 ms    | ~0.3 s      | ~358 s
2x    |  40,000    | 37 ms    | 37 ms    | ~0.8 s      | ~1,364 s
3x    |  60,000    | 57 ms    | 51 ms    | ~1.5 s      | ~4,424 s
4x    |  80,000    | 73 ms    | 68 ms    | ~2.3 s      | ~9,989 s
5x    | 100,000    | 99 ms    | 92 ms    | ~3.3 s      | ~20,014 s

Note:
Trino measurements are adjusted to remove JVM startup overhead (~2.4s
measured via SELECT 1).

Observations

Match success (Test 1)

Both systems scale approximately linearly because the entire segment is
consumed in a single pass after a successful match.

PostgreSQL shows lower absolute latency.

Match failure (Test 2)

PostgreSQL maintains near-linear scaling.

This appears to be due to its NFA implementation combined with Context
Absorption, which discards redundant matching contexts when they reach the
same NFA state but start later in the input.

Example:

Ctx1 start=0 length=2
Ctx2 start=1 length=1

Ctx1 subsumes Ctx2, so Ctx2 is discarded.

This keeps the number of active contexts small and prevents quadratic
growth.

Measured scaling:

17 ms → 92 ms (1x → 5x)

which is consistent with O(n).

Trino also uses an NFA approach but appears to lack Context Absorption or a
similar optimization.

As a result, it explores matching contexts from all possible start
positions, leading to rapidly increasing backtracking cost.

Measured scaling:

358 s → 20,014 s (1x → 5x)

This exceeds theoretical O(n²) growth and appears closer to O(n²)–O(n³) in
practice, likely compounded by JVM memory and GC overhead.

Summary

Correctness

PostgreSQL RPR and Trino MATCH_RECOGNIZE produce identical matching results
across all tested scales.

Performance

Match success:
Both systems scale roughly linearly.

Match failure:
PostgreSQL maintains O(n) scaling while Trino shows O(n²) or worse behavior.

At the largest tested scale:

PostgreSQL : ~92 ms
Trino : ~20,014 s (≈ 5.6 hours)

PostgreSQL is therefore approximately 217,000× faster in this scenario.

Conclusion

Both systems use NFA-based pattern matching.

The key difference appears to be Context Absorption in PostgreSQL, which
removes redundant matching contexts and guarantees linear scaling even when
patterns fail to match.

This optimization prevents the non-linear performance degradation typically
associated with row pattern recognition.

Best regards

SungJun

2026년 3월 9일 (월) PM 2:22, Tatsuo Ishii <ishii@postgresql.org>님이 작성:

> Hi Henson,
>
> >> Excellnt findings!  BTW, I realized that we cannot use $1 of function
> >> in PATTERN clause like: A{$1}.
> >>
> >> ERROR:  42601: syntax error at or near "$1"
> >> LINE 10:         PATTERN (A{$1})
> >>                             ^
> >> LOCATION:  scanner_yyerror, scan.l:1211
> >>
> >> Should we document somewhere?
> >>
> >
> > The PATTERN quantifier {n} only accepts Iconst (integer literal) in the
> > grammar.  When a host variable or function parameter is used (e.g.,
> > A{$1}), the user gets a generic syntax error.
>
> Ok.
>
> > Oracle accepts broader syntax and validates later, producing an error
> > at a later stage rather than a syntax error at parse time.
> >
> > PostgreSQL itself already has precedent for this pattern -- in fact,
> > within the same window clause, frame offset (ROWS/RANGE/GROUPS) accepts
> > a_expr in the grammar and then rejects variables in parse analysis via
> > transformFrameOffset() -> checkExprIsVarFree().
> >
> > I'd lean against documenting this.  The SQL standard already defines
> > the quantifier bound as <unsigned integer literal>, so there is nothing
> > beyond the standard to call out, and documenting what is *not* allowed
> > tends to raise questions that wouldn't otherwise occur to users.
> >
> > Rather, I think accepting a broader grammar and validating later would
> > be the more appropriate response, producing a descriptive error like:
> >
> >   "argument of bounded quantifier must be an integer literal"
> >
> > I can either include this in the current patch set or handle it as a
> > separate follow-up after the main series is committed.  What do you
> > think?
>
> I think handing it as a separate follow-up after the commit is enough
> unless other developers complain.
>
> Best regards,
> --
> Tatsuo Ishii
> SRA OSS K.K.
> English: http://www.sraoss.co.jp/index_en/
> Japanese:http://www.sraoss.co.jp
>

  264. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-11T10:11:17Z

    Hi SungJun,

> Hi hackers,
> 
> I ran a cross-validation test comparing PostgreSQL RPR (current patch) and
> Trino MATCH_RECOGNIZE to verify both result correctness and performance
> characteristics across different scales.

Thank you for the excellent report! I am glad to see that PostgreSQL
RPR is much faster than Trino, especially in the match failure cases.

Is it possible to share the data generation script and the query for
PostgreSQL so that I could locally perform the tests?

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  265. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-03-11T10:32:15Z

    Hi SungJun,

Thanks for the cross-validation report.

This appears to be due to its NFA implementation combined with Context
> Absorption,
>
which discards redundant matching contexts when they reach the same NFA
> state
>
but start later in the input.
>
> Example:
>
> Ctx1 start=0 length=2
> Ctx2 start=1 length=1
>
> Ctx1 subsumes Ctx2, so Ctx2 is discarded.
>
> This keeps the number of active contexts small and prevents quadratic
> growth.
>

Your description is accurate. An earlier-started context subsumes
all matches of a later-started context (monotonicity principle),
so the newer one can be safely removed.

However, Context Absorption is not universally applicable. It
requires careful analysis of several factors, and we continue to
refine the conditions under which absorption can be safely applied.

The current scope of absorption is determined by three factors:

1. Pattern structure:
   Currently supported:
   - Simple unbounded quantifiers: A+, (A B)+
   - Leading unbounded quantifier in group: (A+ B)+
   - First iteration of nested unbounded alternation (e.g., ((A+)|(B+))+)
   Planned improvements:
   - Patterns with a prefix before the repeating group (e.g., A (B+) C)
   - Fixed-length iteration groups (e.g., (A B{2})+ where each
     iteration consumes exactly 3 rows)

2. Frame specification:
   - AFTER MATCH SKIP TO NEXT ROW is not eligible (only
     SKIP PAST LAST ROW)
   - Frame end must be UNBOUNDED FOLLOWING

3. DEFINE clause evaluation:
   - Absorption relies on the monotonicity assumption that DEFINE
     conditions produce the same result regardless of starting
     position. Currently DEFINE uses simple boolean conditions
     where this holds. However, future additions such as
     CLASSIFIER() or running aggregates (e.g., RUNNING SUM())
     may introduce context-dependent evaluation that breaks
     this invariance, requiring absorption to be disabled for
     those cases.

That said, many practically useful patterns fall within these
conditions. Here are examples from our regression tests using
synthetic stock data:

  V-shape recovery (decline then rise):
    PATTERN (DECLINE{4,} RISE{4,})
    DEFINE
      DECLINE AS price < PREV(price),
      RISE AS price > PREV(price)

  W-bottom (decline, bounce, re-decline, recovery):
    PATTERN (DECLINE{3,} BOUNCE{3,} DIP{3,} RECOVER{3,})
    DEFINE
      DECLINE AS price < PREV(price),
      BOUNCE AS price > PREV(price),
      DIP AS price < PREV(price),
      RECOVER AS price > PREV(price)

  Consolidation then breakout:
    PATTERN (FLAT{5,} BREAKOUT)
    DEFINE
      FLAT AS price BETWEEN PREV(price) * 0.98 AND PREV(price) * 1.02,
      BREAKOUT AS price > PREV(price) * 1.05

  Price-volume divergence (bearish signal):
    PATTERN (INIT DIVERGE{3,})
    DEFINE
      DIVERGE AS price > PREV(price) AND volume < PREV(volume)

  Volume climax reversal:
    PATTERN (RALLY{3,} CLIMAX SELLOFF{2,})
    DEFINE
      RALLY AS price > PREV(price),
      CLIMAX AS volume > PREV(volume) * 1.5,
      SELLOFF AS price < PREV(price)

All of these use unbounded quantifiers with position-invariant
DEFINE conditions -- exactly the cases where Context Absorption
guarantees O(n) scaling. So despite the constraints, it remains
a powerful optimization for the workloads where RPR is most
commonly applied.

Best regards,
Henson

  266. Re: Row pattern recognition

    Zsolt Parragi <zsolt.parragi@percona.com> — 2026-03-11T22:39:58Z

    Hello

+ /*
+ * Make sure that the row pattern definition search condition is a boolean
+ * expression.
+ */
+ foreach_ptr(TargetEntry, te, defineClause)
+ (void) coerce_to_boolean(pstate, (Node *) te->expr, "DEFINE");

Isn't this incorrect? I think it should update te->expr, as currently
it is possible to construct queries where this produces unexpected
results.

CREATE TYPE truthyint AS (v int);

CREATE FUNCTION truthyint_to_bool(truthyint) RETURNS boolean AS $$
  SELECT ($1).v <> 0;
$$ LANGUAGE SQL IMMUTABLE STRICT;

CREATE CAST (truthyint AS boolean)
  WITH FUNCTION truthyint_to_bool(truthyint)
  AS ASSIGNMENT;

CREATE TABLE test_coerce (id serial, val truthyint);
INSERT INTO test_coerce VALUES
  (1, ROW(1)),
  (2, ROW(0)),
  (3, ROW(5)),
  (4, ROW(0));

SELECT id, val, COUNT(*) OVER w AS cnt
FROM test_coerce
WINDOW w AS (
    ORDER BY id
    ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
    PATTERN (A+)
    DEFINE A AS val
)
ORDER BY id;

Same query provides the correct result with a table that has an actual
boolean column.

  267. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-03-11T23:52:08Z

    Hi Zsolt, Tatsuo,

+ foreach_ptr(TargetEntry, te, defineClause)
> + (void) coerce_to_boolean(pstate, (Node *) te->expr, "DEFINE");
>
> Isn't this incorrect? I think it should update te->expr, as currently
> it is possible to construct queries where this produces unexpected
> results.


Good catch, Zsolt. You're right — the return value of
coerce_to_boolean() must be assigned back to te->expr, otherwise
any implicit cast (e.g., a user-defined type with an assignment
cast to boolean) is silently discarded.

The fix is straightforward:

    foreach_ptr(TargetEntry, te, defineClause)
        te->expr = (Expr *) coerce_to_boolean(pstate, (Node *) te->expr,
"DEFINE");

I've confirmed that your example query produces incorrect results
without the fix (the truthyint value is evaluated as-is without
the cast) and correct results with it.

Patch 13 is attached with the fix and a regression test case based
on your example.

By the way, thank you for joining the TDE hooks discussion when
I was just getting started as a contributor — that meant a lot.
Great work on pg_tde at Percona, and rooting for the extensible
SMGR effort as well. Hope it's all going well!

Tatsuo, please review when you get a chance.

Regards,
Henson

  268. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-12T00:21:25Z

    Hi Zsolt,

Thanks for the report and the test case!

> Good catch, Zsolt. You're right ― the return value of
> coerce_to_boolean() must be assigned back to te->expr, otherwise
> any implicit cast (e.g., a user-defined type with an assignment
> cast to boolean) is silently discarded.
> 
> The fix is straightforward:
> 
>     foreach_ptr(TargetEntry, te, defineClause)
>         te->expr = (Expr *) coerce_to_boolean(pstate, (Node *) te->expr,
> "DEFINE");
> 
> I've confirmed that your example query produces incorrect results
> without the fix (the truthyint value is evaluated as-is without
> the cast) and correct results with it.
> 
> Patch 13 is attached with the fix and a regression test case based
> on your example.

Yeah, current patch needs to be fixed. Question is, the output of the
expression of DEFINE clause must be a strict boolean or, it is allowed
to accept an expression coercive to boolean?

If we prefer the former, we should use exprType() instead.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  269. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-03-12T00:39:57Z

    2026년 3월 12일 (목) AM 9:21, Tatsuo Ishii <ishii@postgresql.org>님이 작성:

> Hi Zsolt,
>
> Thanks for the report and the test case!
>
> > Good catch, Zsolt. You're right ― the return value of
> > coerce_to_boolean() must be assigned back to te->expr, otherwise
> > any implicit cast (e.g., a user-defined type with an assignment
> > cast to boolean) is silently discarded.
> >
> > The fix is straightforward:
> >
> >     foreach_ptr(TargetEntry, te, defineClause)
> >         te->expr = (Expr *) coerce_to_boolean(pstate, (Node *) te->expr,
> > "DEFINE");
> >
> > I've confirmed that your example query produces incorrect results
> > without the fix (the truthyint value is evaluated as-is without
> > the cast) and correct results with it.
> >
> > Patch 13 is attached with the fix and a regression test case based
> > on your example.
>
> Yeah, current patch needs to be fixed. Question is, the output of the
> expression of DEFINE clause must be a strict boolean or, it is allowed
> to accept an expression coercive to boolean?
>
> If we prefer the former, we should use exprType() instead.
>
> Best regards,
> --
> Tatsuo Ishii
> SRA OSS K.K.
> English: http://www.sraoss.co.jp/index_en/
> Japanese:http://www.sraoss.co.jp
>

  270. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-03-12T00:44:29Z

    Hi Tatsuo,

Sorry about the empty reply — sent it by accident.

Yeah, current patch needs to be fixed. Question is, the output of the
> expression of DEFINE clause must be a strict boolean or, it is allowed
> to accept an expression coercive to boolean?
>
> If we prefer the former, we should use exprType() instead.
>

Good question. Both the SQL standard and PostgreSQL's existing
behavior point toward allowing coercion.

In the SQL standard (ISO/IEC 19075-5), DEFINE specifies a "Boolean
condition" for each pattern variable. The standard does not suggest
a stricter type requirement on DEFINE than on other boolean contexts
like WHERE or HAVING.

PostgreSQL's WHERE clause already accepts implicit casts to boolean
via coerce_to_boolean(). You can verify this with Zsolt's setup:

    CREATE TYPE truthyint AS (v int);
    CREATE FUNCTION truthyint_to_bool(truthyint) RETURNS boolean AS $$
      SELECT ($1).v <> 0;
    $$ LANGUAGE SQL IMMUTABLE STRICT;
    CREATE CAST (truthyint AS boolean)
      WITH FUNCTION truthyint_to_bool(truthyint)
      AS ASSIGNMENT;

    CREATE TABLE test_coerce (id serial, val truthyint);
    INSERT INTO test_coerce VALUES
      (1, ROW(1)), (2, ROW(0)), (3, ROW(5)), (4, ROW(0));

    SELECT id, val FROM test_coerce WHERE val ORDER BY id;
    -- returns rows 1 and 3 (where val casts to true)

As Zsolt noted, the same query works correctly with an actual
boolean column. The issue is specifically that the implicit cast
is not being applied.

All other boolean contexts in the parser (WHERE, HAVING, JOIN/ON,
WHEN, etc.) use coerce_to_boolean() and assign the result back.
I think DEFINE should be consistent with these — it would be
surprising if `DEFINE A AS val` rejected a type that `WHERE val`
accepts.

Regards,
Henson

  271. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-12T01:55:00Z

    > Yeah, current patch needs to be fixed. Question is, the output of the
>> expression of DEFINE clause must be a strict boolean or, it is allowed
>> to accept an expression coercive to boolean?
>>
>> If we prefer the former, we should use exprType() instead.
>>
> 
> Good question. Both the SQL standard and PostgreSQL's existing
> behavior point toward allowing coercion.
> 
> In the SQL standard (ISO/IEC 19075-5), DEFINE specifies a "Boolean
> condition" for each pattern variable. The standard does not suggest
> a stricter type requirement on DEFINE than on other boolean contexts
> like WHERE or HAVING.
> 
> PostgreSQL's WHERE clause already accepts implicit casts to boolean
> via coerce_to_boolean(). You can verify this with Zsolt's setup:
> 
>     CREATE TYPE truthyint AS (v int);
>     CREATE FUNCTION truthyint_to_bool(truthyint) RETURNS boolean AS $$
>       SELECT ($1).v <> 0;
>     $$ LANGUAGE SQL IMMUTABLE STRICT;
>     CREATE CAST (truthyint AS boolean)
>       WITH FUNCTION truthyint_to_bool(truthyint)
>       AS ASSIGNMENT;
> 
>     CREATE TABLE test_coerce (id serial, val truthyint);
>     INSERT INTO test_coerce VALUES
>       (1, ROW(1)), (2, ROW(0)), (3, ROW(5)), (4, ROW(0));
> 
>     SELECT id, val FROM test_coerce WHERE val ORDER BY id;
>     -- returns rows 1 and 3 (where val casts to true)
> 
> As Zsolt noted, the same query works correctly with an actual
> boolean column. The issue is specifically that the implicit cast
> is not being applied.
> 
> All other boolean contexts in the parser (WHERE, HAVING, JOIN/ON,
> WHEN, etc.) use coerce_to_boolean() and assign the result back.
> I think DEFINE should be consistent with these ― it would be
> surprising if `DEFINE A AS val` rejected a type that `WHERE val`
> accepts.

Ok. we should be consistent with WHERE etc. unless the standard
requres DEFINE stricter. But if the tle is in the target list
(possibly with resjunk attribute), it could be different from the one
in the DEFINE because of the casting by coerce_to_boolean(). i.e.:

- tle in the target list has no cast node
- tle in the DEFINE may have cast node

I don't know how this could affect subsequent RPR process but I would
like to keep the prerequisites that tle in DEINE and the traget list
is identical.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  272. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-03-13T00:53:39Z

    Hi Tatsuo,

Thank you for your thoughtful feedback on the tle identity concern.

Ok. we should be consistent with WHERE etc. unless the standard
> requres DEFINE stricter. But if the tle is in the target list
> (possibly with resjunk attribute), it could be different from the one
> in the DEFINE because of the casting by coerce_to_boolean(). i.e.:
>
> - tle in the target list has no cast node
> - tle in the DEFINE may have cast node
>
> I don't know how this could affect subsequent RPR process but I would
> like to keep the prerequisites that tle in DEINE and the traget list
> is identical.
>

> is identical.

That is a very good point regarding tle identity.

On the implementation side, the DEFINE clause TargetEntries are
already deep-copied via copyObject in the parser (parse_rpr.c),
so adding a cast node to the DEFINE tle does not affect the
target list tle -- they are independent copies at that point.

On the semantic side, the two serve different purposes: the target
list produces output columns (where the user expects the original
type), while the DEFINE expression is evaluated as a boolean for
pattern matching. This is the same pattern as WHERE, where the
WHERE expression may carry a cast node but the target list for the
same column does not.

That said, if you would prefer to keep them structurally identical,
an alternative would be to reject non-boolean types in DEFINE with
a clear error (using exprType() check). However, this would be
stricter than WHERE/HAVING behavior, which might surprise users.

I would suggest that allowing coercion is the right choice for
consistency, but I am happy to go with whichever approach you
prefer.


On a separate topic, I have found a server crash bug in the current
PREV/NEXT implementation. Nesting navigation functions causes the
backend to crash:

    CREATE TABLE t (id int, price int);
    INSERT INTO t VALUES (1,10),(2,20),(3,15),(4,25),(5,30);

    SELECT * FROM t WINDOW w AS (
        ORDER BY id
        ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
        PATTERN (A B+)
        DEFINE B AS prev(prev(b.price)) > 0
    );
    -- server process crashes

The SQL standard (ISO/IEC 19075-5, Section 5.6.2) prohibits nesting
PREV/NEXT within PREV/NEXT. The first argument of PREV/NEXT must
contain a column reference, not another navigation function call.
Currently there is no parser-level check for this, so the nested
call passes through to the executor and crashes.

I have attached a test file (rpr_nav_nesting_test.txt) containing
6 test cases that cover the prohibited nesting patterns:

  - PREV(PREV(price))        direct same-kind nesting
  - NEXT(NEXT(price))        direct same-kind nesting
  - PREV(NEXT(price))        cross nesting
  - NEXT(PREV(price))        cross nesting
  - PREV(ABS(PREV(price)))   nesting hidden inside another function
  - NEXT(ABS(NEXT(price)))   nesting hidden inside another function

All of these currently crash the server. After a fix, they should
all produce a syntax error.


Finally, while reviewing the standard and the current
PREV/NEXT implementation, I have some thoughts on the
design.

The current approach uses all three ExprContext slots
(scan/outer/inner) with varno rewriting, which fixes offsets at
+/-1. The standard requires variable offsets like PREV(price, 3),
and the three-slot model cannot support this.

I think a simpler approach would be to use just one slot
(ecxt_outertuple) and swap it during expression evaluation:

  For `price > PREV(price, 2)`:
        ^^^^    ^^^^  ^^^^^
         |       |      |
         v       v      v
    1. [price]       read from current slot   --> datum_a
    2. [PREV( ,2)]   swap slot to (currentpos - 2)
    3.   [price]     read from swapped slot   --> datum_b
    4. [)]           restore slot
    5. [>]           evaluate datum_a > datum_b

Since step 1 already extracts the value as a Datum before the
swap in step 2, the result is safe. No varno rewriting is needed
-- all column reads just use the one slot, and the swap/restore
controls which row they see.

One thing I noticed is that the SQL standard explicitly requires
all column references inside a navigation function to be qualified
by the same row pattern variable (Section 5.2):

  "All row pattern column references in an aggregate or row
   pattern navigation operation are qualified by the same row
   pattern variable."

This means the navigation argument is always evaluated in a single
row context -- which is exactly what makes the one-slot swap model
work. It seems like the standard may have been designed with this
kind of implementation in mind.

With this approach, the existing varno rewriting infrastructure and
the extra slots for PREV/NEXT would no longer be needed, which
should simplify the code considerably.

I also believe this design would scale well to future extensions.
Variable offsets (PREV(price, N)) would work naturally, and when
we add FIRST/LAST navigation or row pattern variable qualifiers,
only the target row calculation would change -- the swap/restore
mechanism itself would remain the same.

What do you think about this approach?

Best regards,
Henson

  273. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-14T07:40:03Z

    Hi Henson,

> Hi Tatsuo,
> 
> Thank you for your thoughtful feedback on the tle identity concern.
> 
> Ok. we should be consistent with WHERE etc. unless the standard
>> requres DEFINE stricter. But if the tle is in the target list
>> (possibly with resjunk attribute), it could be different from the one
>> in the DEFINE because of the casting by coerce_to_boolean(). i.e.:
>>
>> - tle in the target list has no cast node
>> - tle in the DEFINE may have cast node
>>
>> I don't know how this could affect subsequent RPR process but I would
>> like to keep the prerequisites that tle in DEINE and the traget list
>> is identical.
>>
> 
>> is identical.
> 
> That is a very good point regarding tle identity.
> 
> On the implementation side, the DEFINE clause TargetEntries are
> already deep-copied via copyObject in the parser (parse_rpr.c),
> so adding a cast node to the DEFINE tle does not affect the
> target list tle -- they are independent copies at that point.
> 
> On the semantic side, the two serve different purposes: the target
> list produces output columns (where the user expects the original
> type), while the DEFINE expression is evaluated as a boolean for
> pattern matching. This is the same pattern as WHERE, where the
> WHERE expression may carry a cast node but the target list for the
> same column does not.
> 
> That said, if you would prefer to keep them structurally identical,
> an alternative would be to reject non-boolean types in DEFINE with
> a clear error (using exprType() check). However, this would be
> stricter than WHERE/HAVING behavior, which might surprise users.
> 
> I would suggest that allowing coercion is the right choice for
> consistency, but I am happy to go with whichever approach you
> prefer.

I have run a SELECT below after applying your patch:

SELECT id, COUNT(*) OVER w AS cnt
FROM test_coerce WHERE val
WINDOW w AS (
    ORDER BY id
    ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
    PATTERN (A+)
    DEFINE A AS val
)
ORDER BY id;

The point with the SELECT is, the target list lacks val column but it
has WHERE clause with a qual "val".  Thus the SELECT's query tree has
TargetEntry with VAR varattno = 2 (val) and resjunk attribute is true,
and "quals" has cast expression on top of VAR varattno = 2. So the
target entry and qual has different expressions. Probably I should not
expect that the expression in target list is identical to the
expression in other places (WHERE, DEFINE).

> I would suggest that allowing coercion is the right choice for

So I Agree with you.

> On a separate topic, I have found a server crash bug in the current
> PREV/NEXT implementation. Nesting navigation functions causes the
> backend to crash:
> 
>     CREATE TABLE t (id int, price int);
>     INSERT INTO t VALUES (1,10),(2,20),(3,15),(4,25),(5,30);
> 
>     SELECT * FROM t WINDOW w AS (
>         ORDER BY id
>         ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>         PATTERN (A B+)
>         DEFINE B AS prev(prev(b.price)) > 0
>     );
>     -- server process crashes
> 
> The SQL standard (ISO/IEC 19075-5, Section 5.6.2) prohibits nesting
> PREV/NEXT within PREV/NEXT. The first argument of PREV/NEXT must
> contain a column reference, not another navigation function call.
> Currently there is no parser-level check for this, so the nested
> call passes through to the executor and crashes.
> 
> I have attached a test file (rpr_nav_nesting_test.txt) containing
> 6 test cases that cover the prohibited nesting patterns:
> 
>   - PREV(PREV(price))        direct same-kind nesting
>   - NEXT(NEXT(price))        direct same-kind nesting
>   - PREV(NEXT(price))        cross nesting
>   - NEXT(PREV(price))        cross nesting
>   - PREV(ABS(PREV(price)))   nesting hidden inside another function
>   - NEXT(ABS(NEXT(price)))   nesting hidden inside another function
> 
> All of these currently crash the server. After a fix, they should
> all produce a syntax error.

I thought PREV, NEXT nesting check was there but apparently I was
wrong.

> Finally, while reviewing the standard and the current
> PREV/NEXT implementation, I have some thoughts on the
> design.
> 
> The current approach uses all three ExprContext slots
> (scan/outer/inner) with varno rewriting, which fixes offsets at
> +/-1. The standard requires variable offsets like PREV(price, 3),
> and the three-slot model cannot support this.
> 
> I think a simpler approach would be to use just one slot
> (ecxt_outertuple) and swap it during expression evaluation:
> 
>   For `price > PREV(price, 2)`:
>         ^^^^    ^^^^  ^^^^^
>          |       |      |
>          v       v      v
>     1. [price]       read from current slot   --> datum_a
>     2. [PREV( ,2)]   swap slot to (currentpos - 2)
>     3.   [price]     read from swapped slot   --> datum_b
>     4. [)]           restore slot
>     5. [>]           evaluate datum_a > datum_b
> 
> Since step 1 already extracts the value as a Datum before the
> swap in step 2, the result is safe. No varno rewriting is needed
> -- all column reads just use the one slot, and the swap/restore
> controls which row they see.
> 
> One thing I noticed is that the SQL standard explicitly requires
> all column references inside a navigation function to be qualified
> by the same row pattern variable (Section 5.2):
> 
>   "All row pattern column references in an aggregate or row
>    pattern navigation operation are qualified by the same row
>    pattern variable."
> 
> This means the navigation argument is always evaluated in a single
> row context -- which is exactly what makes the one-slot swap model
> work. It seems like the standard may have been designed with this
> kind of implementation in mind.
> 
> With this approach, the existing varno rewriting infrastructure and
> the extra slots for PREV/NEXT would no longer be needed, which
> should simplify the code considerably.
> 
> I also believe this design would scale well to future extensions.
> Variable offsets (PREV(price, N)) would work naturally, and when
> we add FIRST/LAST navigation or row pattern variable qualifiers,
> only the target row calculation would change -- the swap/restore
> mechanism itself would remain the same.
> 
> What do you think about this approach?

Looks great. Do you have any idea how to let the existing ExecEvalExpr
handle the swap/restore mechanism?

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  274. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-03-14T14:14:26Z

    Hi Tatsuo,

Looks great. Do you have any idea how to let the existing ExecEvalExpr
> handle the swap/restore mechanism?



Yes. I think the key insight is that ExecEvalExpr compiles expressions into
a flat array of ExprEvalStep operations, which makes the swap/restore
natural
to implement without any changes to the ExecEvalExpr caller interface.

I should also say that this approach was shaped by the constraints you
identified early on -- the three-slot model's limitations and the need
for variable offsets. Without that clear problem framing, I would not
have thought in this direction.

The idea is to introduce two new ExprEvalOp steps:

  EEOP_RPR_NAV_SET     -- save current slot, swap to target row
  EEOP_RPR_NAV_RESTORE -- restore original slot

These two steps bracket the inner expression steps in the flat array.

For `price > PREV(price)`:
  1. EEOP_OUTER_VAR      -- fetch price from current slot --> datum_a
  2. EEOP_RPR_NAV_SET    -- save current slot, swap to (currentpos - 1)
  3. EEOP_OUTER_VAR      -- fetch price from swapped slot --> datum_b
  4. EEOP_RPR_NAV_RESTORE -- restore original slot
  5. EEOP_GT             -- evaluate datum_a > datum_b
  6. EEOP_DONE

For `price > PREV(price, 3)` (with offset):
  1. EEOP_OUTER_VAR      -- fetch price from current slot --> datum_a
  2. EEOP_CONST          -- evaluate offset constant 3 --> datum_off
  3. EEOP_RPR_NAV_SET    -- save current slot, swap to (currentpos - 3)
  4. EEOP_OUTER_VAR      -- fetch price from swapped slot --> datum_b
  5. EEOP_RPR_NAV_RESTORE -- restore original slot
  6. EEOP_GT             -- evaluate datum_a > datum_b
  7. EEOP_DONE

For `price < NEXT(price)`:
  1. EEOP_OUTER_VAR      -- fetch price from current slot --> datum_a
  2. EEOP_RPR_NAV_SET    -- save current slot, swap to (currentpos + 1)
  3. EEOP_OUTER_VAR      -- fetch price from swapped slot --> datum_b
  4. EEOP_RPR_NAV_RESTORE -- restore original slot
  5. EEOP_LT             -- evaluate datum_a < datum_b
  6. EEOP_DONE

ExprState holds the flat array of ExprEvalSteps, and each step carries
its own payload. The RPR navigation steps would store a pointer to the
window execution state in their payload, so that they can access the
current row position and the tuplestore at evaluation time. Since
PREV/NEXT are only permitted in the DEFINE clause, the window execution
state is always valid when these steps are evaluated, which eliminates
the risk of accessing stale or invalid state.

The EEOP_RPR_NAV_SET step would:
  1. Save the current slot for later restore
  2. Compute the target position from currentpos and the direction offset
  3. Fetch the corresponding TupleTableSlot from the tuplestore
  4. Replace econtext->ecxt_outertuple with the target slot

The EEOP_RPR_NAV_RESTORE step would:
  1. Retrieve the previously saved slot
  2. Restore econtext->ecxt_outertuple to the original slot

This design also fits well with the future roadmap for FIRST/LAST
navigation.
Once the match history infrastructure is in place, FIRST/LAST can look
up the first or last row matched by a given pattern variable from the
history, compute the target position from that, and pass it to
EEOP_RPR_NAV_SET -- the save/restore mechanism itself stays the same.

I would like to work on an experimental implementation of this approach,
including the parser-level check to reject nested PREV/NEXT calls as
required by the standard. The nesting check will be part of the same
implementation rather than a separate fix, as the parser structure may
need to be adjusted as part of the redesign anyway.

Since the expression compilation infrastructure is not an area I am
deeply familiar with, it may take some time to get right. I would
appreciate your patience in reviewing it when it is ready.

Best regards,
Henson

  275. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-15T09:59:36Z

    Hi Henson,

> Yes. I think the key insight is that ExecEvalExpr compiles expressions into
> a flat array of ExprEvalStep operations, which makes the swap/restore
> natural
> to implement without any changes to the ExecEvalExpr caller interface.
> 
> I should also say that this approach was shaped by the constraints you
> identified early on -- the three-slot model's limitations and the need
> for variable offsets. Without that clear problem framing, I would not
> have thought in this direction.

I am not an expert of this area but I have some questions.

> The idea is to introduce two new ExprEvalOp steps:
> 
>   EEOP_RPR_NAV_SET     -- save current slot, swap to target row
>   EEOP_RPR_NAV_RESTORE -- restore original slot
> 
> These two steps bracket the inner expression steps in the flat array.
> 
> For `price > PREV(price)`:
>   1. EEOP_OUTER_VAR      -- fetch price from current slot --> datum_a
>   2. EEOP_RPR_NAV_SET    -- save current slot, swap to (currentpos - 1)
>   3. EEOP_OUTER_VAR      -- fetch price from swapped slot --> datum_b
>   4. EEOP_RPR_NAV_RESTORE -- restore original slot
>   5. EEOP_GT             -- evaluate datum_a > datum_b

I was not able to find the definition for EEOP_GT. Is this a new one?

> The EEOP_RPR_NAV_SET step would:
>   1. Save the current slot for later restore
>   2. Compute the target position from currentpos and the direction offset
>   3. Fetch the corresponding TupleTableSlot from the tuplestore

I am afraid ExecEvalExpr does not allow this because ExecEvalExpr does
not accept arguments like WindowObject or WindowAggState that are
necessary for accessing the tuplestore created for the Window
aggregate node.

That is (ExecEvalExpr doe not accept arguments like WindowObject or
WindowAggState) the reason why EEOP_WINDOW_FUNC is implemented like this:

			/*
			 * Like Aggref, just return a precomputed value from the econtext.
			 */
			WindowFuncExprState *wfunc = op->d.window_func.wfstate;

			Assert(econtext->ecxt_aggvalues != NULL);

			*op->resvalue = econtext->ecxt_aggvalues[wfunc->wfuncno];
			*op->resnull = econtext->ecxt_aggnulls[wfunc->wfuncno];

so that ExecEvalExpr can access the window function's result from
op->resvalue and op->resnull.  See eval_windowfunction for more
details.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  276. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-03-16T05:39:47Z

    Hi Tatsuo,

Thank you for the careful review and excellent questions.

I was not able to find the definition for EEOP_GT. Is this a new one?
>

You're right -- EEOP_GT does not exist. I oversimplified the example.

In PostgreSQL, comparison operators like `>` are not dedicated opcodes.
They are dispatched through EEOP_FUNCEXPR variants as regular function
calls. For example, `price > PREV(price)`:

  - The parser resolves `>` to the appropriate pg_operator entry
  - The expression compiler emits an EEOP_FUNCEXPR variant
  - At runtime, op->d.func.fn_addr points to the operator function,

I am afraid ExecEvalExpr does not allow this because ExecEvalExpr does
> not accept arguments like WindowObject or WindowAggState that are
> necessary for accessing the tuplestore created for the Window
> aggregate node.


This is the key question. You are right that ExecEvalExpr's signature
does not accept WindowAggState, and I do not intend to change it.

The approach is to pass WindowAggState through the step's own payload,
the same way EEOP_WINDOW_FUNC carries WindowFuncExprState in
op->d.window_func.wfstate. Each ExprEvalStep has a union of per-opcode
payload structs. A new payload for RPR navigation would look like:

    struct
    {
        WindowAggState *winstate;   /* access to tuplestore */
        int64       offset;         /* signed offset: PREV=-1, NEXT=+1 */
    }           rpr_nav;

The expression compiler (ExecInitExprRec) populates this when it
encounters a PREV/NEXT node, storing the pointer to the enclosing
WindowAggState. At runtime, EEOP_RPR_NAV_SET retrieves it from the
payload and uses it to fetch the target row:

    /* pseudo-code for EEOP_RPR_NAV_SET */
    winstate = op->d.rpr_nav.winstate;
    /* 1. save current slot into winstate for later restore */
    winstate->saved_outertuple = econtext->ecxt_outertuple;
    /* 2. compute target position */
    target_pos = winstate->currentpos + op->d.rpr_nav.offset;
    /* 3. fetch target row from tuplestore via winstate */
    target_slot = fetch_row_from_tuplestore(winstate, target_pos);
    /* 4. swap */
    econtext->ecxt_outertuple = target_slot;

    /* pseudo-code for EEOP_RPR_NAV_RESTORE */
    winstate = op->d.rpr_nav.winstate;
    econtext->ecxt_outertuple = winstate->saved_outertuple;

The key difference from EEOP_WINDOW_FUNC: existing steps read
precomputed values, but PREV/NEXT requires fetching a *different*
row at runtime -- because the target depends on the current row
position during pattern matching, which is not known in advance.
The mechanism for passing state is the same (step payload), but
the runtime behavior is a departure from the existing pattern.

I have examined the alternatives carefully:

  - The precomputation approach (EEOP_WINDOW_FUNC style) does not apply
    here: there is no fixed set of values to precompute, because the
    target row depends on the current position during pattern matching.

  - The three-slot model (current implementation) cannot support variable
    offsets like PREV(price, 3), and requires varno rewriting that adds
    complexity elsewhere.

  - A callback-based approach (registering a fetch function in econtext)
    would still require passing WindowAggState somehow, just through a
    different indirection.

The step payload approach is the only path I have found that:

  - Leaves ExecEvalExpr's signature unchanged
  - Eliminates varno rewriting entirely
  - Supports variable offsets naturally
  - Extends to FIRST/LAST navigation later

If anyone sees a cleaner path, I would genuinely welcome the
discussion.

I plan to have an experimental implementation ready before next week.
The initial version will pass WindowAggState broadly; we can
narrow the interface to the minimum required during review.
Let's continue the discussion with actual code.

Best regards,
Henson

  277. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-16T06:47:32Z

    > I was not able to find the definition for EEOP_GT. Is this a new one?
>>
> 
> You're right -- EEOP_GT does not exist. I oversimplified the example.
> 
> In PostgreSQL, comparison operators like `>` are not dedicated opcodes.
> They are dispatched through EEOP_FUNCEXPR variants as regular function
> calls. For example, `price > PREV(price)`:
> 
>   - The parser resolves `>` to the appropriate pg_operator entry
>   - The expression compiler emits an EEOP_FUNCEXPR variant
>   - At runtime, op->d.func.fn_addr points to the operator function,

Ok.

> I am afraid ExecEvalExpr does not allow this because ExecEvalExpr does
>> not accept arguments like WindowObject or WindowAggState that are
>> necessary for accessing the tuplestore created for the Window
>> aggregate node.
> 
> 
> This is the key question. You are right that ExecEvalExpr's signature
> does not accept WindowAggState, and I do not intend to change it.
> 
> The approach is to pass WindowAggState through the step's own payload,
> the same way EEOP_WINDOW_FUNC carries WindowFuncExprState in
> op->d.window_func.wfstate. Each ExprEvalStep has a union of per-opcode
> payload structs. A new payload for RPR navigation would look like:
> 
>     struct
>     {
>         WindowAggState *winstate;   /* access to tuplestore */
>         int64       offset;         /* signed offset: PREV=-1, NEXT=+1 */
>     }           rpr_nav;
> 
> The expression compiler (ExecInitExprRec) populates this when it
> encounters a PREV/NEXT node, storing the pointer to the enclosing
> WindowAggState. At runtime, EEOP_RPR_NAV_SET retrieves it from the
> payload and uses it to fetch the target row:
> 
>     /* pseudo-code for EEOP_RPR_NAV_SET */
>     winstate = op->d.rpr_nav.winstate;
>     /* 1. save current slot into winstate for later restore */
>     winstate->saved_outertuple = econtext->ecxt_outertuple;
>     /* 2. compute target position */
>     target_pos = winstate->currentpos + op->d.rpr_nav.offset;
>     /* 3. fetch target row from tuplestore via winstate */
>     target_slot = fetch_row_from_tuplestore(winstate, target_pos);
>     /* 4. swap */
>     econtext->ecxt_outertuple = target_slot;
> 
>     /* pseudo-code for EEOP_RPR_NAV_RESTORE */
>     winstate = op->d.rpr_nav.winstate;
>     econtext->ecxt_outertuple = winstate->saved_outertuple;
> 
> The key difference from EEOP_WINDOW_FUNC: existing steps read
> precomputed values, but PREV/NEXT requires fetching a *different*
> row at runtime -- because the target depends on the current row
> position during pattern matching, which is not known in advance.
> The mechanism for passing state is the same (step payload), but
> the runtime behavior is a departure from the existing pattern.
> 
> I have examined the alternatives carefully:
> 
>   - The precomputation approach (EEOP_WINDOW_FUNC style) does not apply
>     here: there is no fixed set of values to precompute, because the
>     target row depends on the current position during pattern matching.
>
>   - The three-slot model (current implementation) cannot support variable
>     offsets like PREV(price, 3), and requires varno rewriting that adds
>     complexity elsewhere.
> 
>   - A callback-based approach (registering a fetch function in econtext)
>     would still require passing WindowAggState somehow, just through a
>     different indirection.
> 
> The step payload approach is the only path I have found that:
> 
>   - Leaves ExecEvalExpr's signature unchanged
>   - Eliminates varno rewriting entirely
>   - Supports variable offsets naturally
>   - Extends to FIRST/LAST navigation later

Thank you for the detailed explanation. I agreed the approach.

> If anyone sees a cleaner path, I would genuinely welcome the
> discussion.
> 
> I plan to have an experimental implementation ready before next week.
> The initial version will pass WindowAggState broadly; we can
> narrow the interface to the minimum required during review.
> Let's continue the discussion with actual code.

Looking forward to reviewing the implementation.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  278. Re: Row pattern recognition

    SungJun Jang <sjjang112233@gmail.com> — 2026-03-17T04:59:24Z

    Hi hackers,

Thank you for the excellent report! I am glad to see that PostgreSQL
> RPR is much faster than Trino, especially in the match failure cases.
>
> Is it possible to share the data generation script and the query for
> PostgreSQL so that I could locally perform the tests?


Here is a self-contained guide to reproduce the RPR ABCD pattern test
locally
using PostgreSQL, Trino, and Oracle.


Requirements:

PostgreSQL 19devel build with the RPR patch applied (local)

Trino and Oracle can be set up via Docker (see README for details):

  https://github.com/assam258-5892/docker-databases

Start all services:

  cd ~/docker-databases && docker compose up -d trino-service oracle-service

Note: docker compose up does not pull images automatically if they are
already
cached locally. To fetch the latest images, run docker compose pull first.

Note: The Oracle image requires an Oracle account. Register at
https://container-registry.oracle.com, accept the license for the database
image, then log in before pulling:

  docker login container-registry.oracle.com

Connect to each database shell:

  Trino:  docker compose exec -it trino-service trino
  Oracle: docker compose exec -it oracle-service sqlplus / as sysdba


Step 1: Create the test table and data

PostgreSQL (1x scale, 20,000 rows):

DROP TABLE IF EXISTS abcd_test;
CREATE TABLE abcd_test AS
SELECT v,
       CASE
           WHEN v % 10000 < 3333 THEN 'A'
           WHEN v % 10000 >= 3333 AND v % 10000 < 6666 THEN 'B'
           WHEN v % 10000 >= 6666 AND v % 10000 < 9999 THEN 'C'
           WHEN v % 10000 = 9999 THEN 'D'
       END AS cat
FROM generate_series(0, 19999) AS v;

ANALYZE abcd_test;


Trino (1x scale, 20,000 rows):

CREATE SCHEMA IF NOT EXISTS memory.test;

DROP TABLE IF EXISTS memory.test.abcd_test;
CREATE TABLE memory.test.abcd_test AS
WITH nums AS (
    SELECT a.v * 10000 + b.v AS v
    FROM UNNEST(sequence(0, 1)) AS a(v)
    CROSS JOIN UNNEST(sequence(0, 9999)) AS b(v)
)
SELECT CAST(v AS INTEGER) AS v,
       CASE
           WHEN v % 10000 < 3333 THEN 'A'
           WHEN v % 10000 >= 3333 AND v % 10000 < 6666 THEN 'B'
           WHEN v % 10000 >= 6666 AND v % 10000 < 9999 THEN 'C'
           WHEN v % 10000 = 9999 THEN 'D'
       END AS cat
FROM nums;

Note: Trino sequence() is limited to 10,000 elements per call, so a CROSS
JOIN
is used. For scale Sx, change sequence(0, 1) to sequence(0, S*2-1).


Oracle (1x scale, 20,000 rows):

DROP TABLE abcd_test PURGE;
CREATE TABLE abcd_test AS
SELECT v,
       CASE
           WHEN MOD(v, 10000) < 3333 THEN 'A'
           WHEN MOD(v, 10000) >= 3333 AND MOD(v, 10000) < 6666 THEN 'B'
           WHEN MOD(v, 10000) >= 6666 AND MOD(v, 10000) < 9999 THEN 'C'
           WHEN MOD(v, 10000) = 9999 THEN 'D'
       END AS cat
FROM (SELECT LEVEL - 1 AS v FROM dual CONNECT BY LEVEL <= 20000);

EXEC DBMS_STATS.GATHER_TABLE_STATS(USER, 'ABCD_TEST');


Verify data distribution (all engines):

SELECT cat, COUNT(*) AS cnt FROM abcd_test GROUP BY cat ORDER BY cat;

Expected (1x): A=6666, B=6666, C=6666, D=2


Step 2: Run Test 1 — A+ B+ C+ D (match expected)

Expected: 2 rows returned (one match per segment)

PostgreSQL:

SELECT match_first, match_last, match_len
FROM (
    SELECT v,
           first_value(v) OVER w AS match_first,
           last_value(v) OVER w AS match_last,
           count(*) OVER w AS match_len
    FROM abcd_test
    WINDOW w AS (
        ORDER BY v
        ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
        AFTER MATCH SKIP PAST LAST ROW
        PATTERN (A+ B+ C+ D)
        DEFINE
            A AS cat = 'A',
            B AS cat = 'B',
            C AS cat = 'C',
            D AS cat = 'D'
    )
) result
WHERE match_len > 0;


Trino / Oracle:

SELECT match_first, match_last, match_len
FROM abcd_test
MATCH_RECOGNIZE (
    ORDER BY v
    MEASURES
        FIRST(v) AS match_first,
        LAST(v) AS match_last,
        COUNT(*) AS match_len
    ONE ROW PER MATCH
    AFTER MATCH SKIP PAST LAST ROW
    PATTERN (A+ B+ C+ D)
    DEFINE
        A AS cat = 'A',
        B AS cat = 'B',
        C AS cat = 'C',
        D AS cat = 'D'
) mr;

Note: Trino uses memory.test.abcd_test as the table name.

Expected result (1x):

match_first | match_last | match_len
------------|------------|----------
          0 |       9999 |     10000
      10000 |      19999 |     10000


Step 3: Run Test 2 — A+ B+ C+ E (match failure)

Expected: 0 rows (E does not exist)

Use the same queries as Test 1 with two changes:

PATTERN: (A+ B+ C+ D) → (A+ B+ C+ E)

DEFINE: cat = 'D' → cat = 'E'

Warning: Trino Test 2 at 1x scale takes approximately 5-6 minutes.


Step 4: Scale up (optional)

Re-create the test table at 2x scale (40,000 rows) and then repeat Step 2
and
Step 3.

PostgreSQL (2x scale, 40,000 rows):

DROP TABLE IF EXISTS abcd_test;
CREATE TABLE abcd_test AS
SELECT v,
       CASE
           WHEN v % 20000 < 6666 THEN 'A'
           WHEN v % 20000 >= 6666 AND v % 20000 < 13332 THEN 'B'
           WHEN v % 20000 >= 13332 AND v % 20000 < 19999 THEN 'C'
           WHEN v % 20000 = 19999 THEN 'D'
       END AS cat
FROM generate_series(0, 39999) AS v;

ANALYZE abcd_test;


Trino (2x scale, 40,000 rows):

CREATE SCHEMA IF NOT EXISTS memory.test;

DROP TABLE IF EXISTS memory.test.abcd_test;
CREATE TABLE memory.test.abcd_test AS
WITH nums AS (
    SELECT a.v * 10000 + b.v AS v
    FROM UNNEST(sequence(0, 3)) AS a(v)
    CROSS JOIN UNNEST(sequence(0, 9999)) AS b(v)
)
SELECT CAST(v AS INTEGER) AS v,
       CASE
           WHEN v % 20000 < 6666 THEN 'A'
           WHEN v % 20000 >= 6666 AND v % 20000 < 13332 THEN 'B'
           WHEN v % 20000 >= 13332 AND v % 20000 < 19999 THEN 'C'
           WHEN v % 20000 = 19999 THEN 'D'
       END AS cat
FROM nums;


Oracle (2x scale, 40,000 rows):

DROP TABLE abcd_test PURGE;
CREATE TABLE abcd_test AS
SELECT v,
       CASE
           WHEN MOD(v, 20000) < 6666 THEN 'A'
           WHEN MOD(v, 20000) >= 6666 AND MOD(v, 20000) < 13332 THEN 'B'
           WHEN MOD(v, 20000) >= 13332 AND MOD(v, 20000) < 19999 THEN 'C'
           WHEN MOD(v, 20000) = 19999 THEN 'D'
       END AS cat
FROM (SELECT LEVEL - 1 AS v FROM dual CONNECT BY LEVEL <= 40000);

EXEC DBMS_STATS.GATHER_TABLE_STATS(USER, 'ABCD_TEST');

Expected (2x): A=13332, B=13332, C=13332, D=4

Then run Step 2 and Step 3 as-is.

Warning: Trino Test 2 at 2x scale takes approximately 20-25 minutes.


Please let me know if you encounter any issues reproducing this.

Best regards
SungJun

  279. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-18T11:41:39Z

    Hi Henson,

Attached is the v45 patches. These are basically your patches
(0001-0013) on top of v44, plus rebase. Major changes/fixes since v44
include:

- Rebase. Due to commits (SQL/PG) rebase was required.

  I had to resolve shift/reduce conflicts while rebasing gram.y. In
  the end, I removed follwing in the v44 patch.
---------------------------------------------------------------------------
 qual_Op:	Op
 					{ $$ = list_make1(makeString($1)); }
 			| OPERATOR '(' any_operator ')'
 					{ $$ = $3; }
+			| '|'
+					{ $$ = list_make1(makeString("|")); }
 		;
--------------------------------------------------------------------------- 

  Also, we don't need the patch to scan.l anymore since the necessary
  changes are already there, thanks to SQL/PGQ commit.
  
- Fix elog message to use lowercase per PostgreSQL convention

  Minor style fix: lowercase the elog error message in
  nodeWindowAgg.c to follow PostgreSQL coding convention.

- Fix RPR reluctant quantifier flag lost during VIEW serialization

  The reluctant flag on quantifiers was lost when a VIEW containing
  RPR was serialized and restored via ruleutils.c.  This patch fixes
  gram.y, rpr.c, ruleutils.c, and parsenodes.h so the flag survives
  the round-trip.

- Expand RPR test coverage and improve test comments

  Reorganizes test cases across rpr.sql, rpr_base.sql, rpr_explain.sql,
  and rpr_nfa.sql.  Moves base functionality tests from rpr.sql to
  rpr_base.sql and NFA-specific tests to rpr_nfa.sql.  Adds better
  section comments throughout.  Duplicate tests in rpr.sql were
  removed.  No reduction in test coverage.

- Keep RPR test objects for pg_upgrade/pg_dump testing

  Adjusts rpr_base.sql and rpr_explain.sql so that test tables and
  views are not dropped at the end of the test.  This allows
  pg_upgrade and pg_dump regression testing to cover RPR objects.

- Disable run condition pushdown for RPR windows
  Revised version of Tatsuo's run condition fix [1].  Existing test
  expected output updated accordingly.

- Disable frame optimization for RPR windows

  Prevents the planner from optimizing the window frame for RPR
  windows.  The frame clause in RPR has different semantics (it
  bounds the pattern search space), so standard frame optimizations
  must be disabled.  Adds EXPLAIN tests to verify.  Please review
  this one -- it's a newly discovered issue.

- Add stock scenario tests for RPR pattern matching

  Adds stock trading scenario tests using realistic synthetic data
  (stock.data with 1632 rows).  Tests V-shape recovery, W-shape,
  consecutive rises, and other common pattern matching use cases.

- Fix zero-min reluctant quantifier to produce zero-length match

  Fixes the bug reported by SungJun [2]: reluctant quantifiers with
  min=0 (A*?, A??) were incorrectly consuming at least one row instead
  of producing a zero-length match.  The NFA can now internally
  distinguish between a zero-length match and no match, so it will
  be ready when additional infrastructure (e.g. MATCH_NUMBER) is
  added.  Now matches Oracle behavior.

- Fix coerce_to_boolean result not applied in DEFINE clause

[1]
https://www.postgresql.org/message-id/20260304.153822.445473532741409674.ishii@postgresql.org
[2]
https://www.postgresql.org/message-id/CAE+cgNiUbKeH1A0PoxV2QjpsoxJLe+pJcGz_gdxwOwu_9zqchw@mail.gmail.com

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  280. Re: Row pattern recognition

    Zsolt Parragi <zsolt.parragi@percona.com> — 2026-03-18T23:06:59Z

    Hello

I found one more bug, the following testcase crashes the server:

CREATE TEMP TABLE t (id int, val text);
INSERT INTO t VALUES
  (1,'A'),(2,'B'),
  (3,'A'),(4,'B'),
  (5,'A'),(6,'B'),
  (7,'A'),(8,'B'),
  (9,'X');

SELECT id, val, count(*) OVER w AS match_count
FROM t
WINDOW w AS (
  ORDER BY id
  ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
  INITIAL
  PATTERN (A B (A B){1,2} A B)
  DEFINE
    A AS val = 'A',
    B AS val = 'B'
);

Where the code does

+ child->min += 1;

It should also increment max if it's not infinity.

if (child->max != RPR_QUANTITY_INF)
    child->max += 1;


+ expr = te->expr;
...
+ attno_map((Node *) expr);

It it okay to mutate an object in the plan cache in the executor?
Wouldn't it be better to first copy it?

> Adds stock trading scenario tests using realistic synthetic data
> (stock.data with 1632 rows). Tests V-shape recovery, W-shape,
> consecutive rises, and other common pattern matching use cases.

The data file is not included in the patchset

+ if (wc->frameOptions & FRAMEOPTION_GROUPS)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("FRAME option GROUP is not permitted when row pattern
recognition is used"),
+ errhint("Use: ROWS instead"),
+ parser_errposition(pstate,
+ windef->frameLocation >= 0 ?
+ windef->frameLocation : windef->location)));

Here and at a few other places, shouldn't this use a different error
code, like ERRCODE_WINDOWING_ERROR?

Also a typo, the error message should say GROUPS.

+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args",
+ func->funcid, nargs);

Isn't this a user triggerable error, which should have a proper error code?

  281. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-19T01:39:30Z

    Thanks for the review!

> I found one more bug, the following testcase crashes the server:
> 
> CREATE TEMP TABLE t (id int, val text);
> INSERT INTO t VALUES
>   (1,'A'),(2,'B'),
>   (3,'A'),(4,'B'),
>   (5,'A'),(6,'B'),
>   (7,'A'),(8,'B'),
>   (9,'X');
> 
> SELECT id, val, count(*) OVER w AS match_count
> FROM t
> WINDOW w AS (
>   ORDER BY id
>   ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>   INITIAL
>   PATTERN (A B (A B){1,2} A B)
>   DEFINE
>     A AS val = 'A',
>     B AS val = 'B'
> );

I just confirmed the crash.

> Where the code does
> 
> + child->min += 1;
> 
> It should also increment max if it's not infinity.
> 
> if (child->max != RPR_QUANTITY_INF)
>     child->max += 1;

Henson, What do you think? 

> + expr = te->expr;
> ...
> + attno_map((Node *) expr);
> 
> It it okay to mutate an object in the plan cache in the executor?
> Wouldn't it be better to first copy it?

Good point. You are right, the plan cache should be read only.
However, Henson is working on a different approach and the code will
not be used any more...

>> Adds stock trading scenario tests using realistic synthetic data
>> (stock.data with 1632 rows). Tests V-shape recovery, W-shape,
>> consecutive rises, and other common pattern matching use cases.
> 
> The data file is not included in the patchset

I forggot to include the data file. Please apply attached patch on top
of v45.

> + if (wc->frameOptions & FRAMEOPTION_GROUPS)
> + ereport(ERROR,
> + (errcode(ERRCODE_SYNTAX_ERROR),
> + errmsg("FRAME option GROUP is not permitted when row pattern
> recognition is used"),
> + errhint("Use: ROWS instead"),
> + parser_errposition(pstate,
> + windef->frameLocation >= 0 ?
> + windef->frameLocation : windef->location)));
> 
> Here and at a few other places, shouldn't this use a different error
> code, like ERRCODE_WINDOWING_ERROR?

Agreed.

> Also a typo, the error message should say GROUPS.
> 
> + elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args",
> + func->funcid, nargs);
> 
> Isn't this a user triggerable error, which should have a proper error code?

Right. Maybe ERRCODE_TOO_MANY_ARGUMENTS?

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  282. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-03-19T02:28:29Z

    Hi Tatsuo, Zsolt,

I just confirmed the crash.
>
> Henson, What do you think?
>

Good catch by Zsolt. Your careful review is really helping a lot.

I've fixed both the prefix merge and suffix merge paths in
mergeGroupPrefixSuffix() to also increment child->max. I'll
include the fix with a regression test in the next patch series.

Regarding the suggested fix:

    if (child->max != RPR_QUANTITY_INF)
        child->max += 1;

While the quantifier value is int (INT32_MAX = RPR_QUANTITY_INF),
making such repetition counts practically impossible, the +1
approach is not semantically equivalent -- it could silently turn
a finite max into infinity. Instead, I took a fallback approach:
skip the merge entirely when min or max would reach
RPR_QUANTITY_INF after increment, consistent with the overflow
checks in mergeConsecutiveVars/Groups.


> Good point. You are right, the plan cache should be read only.
> However, Henson is working on a different approach and the code will
> not be used any more...
>

Yes, I'm currently working on a slot-based approach (1-slot
PREV/NEXT) that won't need attno_map, so the plan cache mutation
issue will be gone.


> I forggot to include the data file. Please apply attached patch on top
> of v45.
>

Got it, thanks. By the way, how about splitting the test patch
into two -- one for sql + data files and another for expected
output files? The single test patch is getting quite large and
cfbot often fails to apply it.

Right. Maybe ERRCODE_TOO_MANY_ARGUMENTS?
>

 I'm currently reworking PREV/NEXT to accept 2 arguments, so this
error handling will change. I'll take care of the proper error
codes along with the GROUPS typo and ERRCODE_WINDOWING_ERROR change.

Best regards,
Henson

  283. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-19T03:15:50Z

    Hi Henson,

>> I forggot to include the data file. Please apply attached patch on top
>> of v45.
>>
> 
> Got it, thanks. By the way, how about splitting the test patch
> into two -- one for sql + data files and another for expected
> output files? The single test patch is getting quite large and
> cfbot often fails to apply it.

Sounds like a good idea.

> Right. Maybe ERRCODE_TOO_MANY_ARGUMENTS?
>>
> 
>  I'm currently reworking PREV/NEXT to accept 2 arguments, so this
> error handling will change. I'll take care of the proper error
> codes along with the GROUPS typo and ERRCODE_WINDOWING_ERROR change.

Ok, thanks.
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  284. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-19T10:10:35Z

    Hi Henson,

>> Got it, thanks. By the way, how about splitting the test patch
>> into two -- one for sql + data files and another for expected
>> output files? The single test patch is getting quite large and
>> cfbot often fails to apply it.
> 
> Sounds like a good idea.

I have made an estimate how large each patch part is.

SQL + data patches:
 src/test/regress/sql/rpr.sql              | 1164 ++++
 src/test/regress/sql/rpr_base.sql         | 3986 +++++++++++++
 src/test/regress/sql/rpr_explain.sql      | 2296 ++++++++
 src/test/regress/sql/rpr_nfa.sql          | 3278 +++++++++++
 src/test/regress/parallel_schedule        |    5 +
 src/test/regress/data/stock.data          | 1632
-----------------------------------------------------
 total					   12361

Expected patches:
 src/test/regress/expected/rpr.out         | 2356 ++++++++
 src/test/regress/expected/rpr_base.out    | 6202 +++++++++++++++++++++
 src/test/regress/expected/rpr_explain.out | 3968 +++++++++++++
 src/test/regress/expected/rpr_nfa.out     | 4340 ++++++++++++++
-----------------------------------------------------
 total					   16866

It seems we will have about 12k (sql and data) and 16k (expected)
patches. Looks nice balance.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  285. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-03-20T15:36:18Z

    Hi Tatsuo,

Sorry for sending these before your next revision of v45 --
I'll be busy with a project proposal for the next couple of
weeks, so I wanted to share them before that.

Here are the patches on top of v45, incorporating the fixes and
improvements I mentioned earlier, plus additional changes from a
thorough code review. There are quite a few patches since I went
ahead without prior discussion -- please review each one individually
and feel free to drop any that you disagree with.

Here's a summary of each patch (to be applied on top of v45):

  0001: Fix mergeGroupPrefixSuffix() max increment
        When absorbing prefix/suffix, child->max was not incremented,
        causing incorrect quantifier bounds. (reported by Zsolt)

  0002: Fix RPR error codes and GROUPS typo
        Use ERRCODE_WINDOWING_ERROR instead of ERRCODE_SYNTAX_ERROR
        for semantic window-frame violations. Fix "GROUP" typo to
        "GROUPS" in frame validation error message. (reported by Zsolt)

  0003: Add check_stack_depth() and CHECK_FOR_INTERRUPTS()
        Add stack depth protection to recursive pattern optimization
        and interrupt checks to NFA engine loops.

  0004: Fix window_last_value set_mark during RPR
        Restore set_mark=true in window_last_value (normal behavior)
        and add an RPR-specific override inside WinGetFuncArgInFrame.
        This keeps the RPR workaround localized rather than changing
        the caller's semantics unconditionally.

  0005: Fix row_is_in_reduced_frame in WINDOW_SEEK_TAIL
        Pass frameheadpos directly to row_is_in_reduced_frame
        instead of frameheadpos+relpos. Currently only last_value()
        calls this path with relpos=0 so no actual bug, but the
        old expression would be incorrect for negative relpos.
        Also add a bounds check for future callers.

  0006: Clarify ST_NONE intent
        Add comment explaining ST_NONE = 0 is the default for
        non-RPR windows.

  0007: Clarify inverse transition optimization comment
        Document why RPR disables inverse transition: the reduced
        frame changes row by row.

  0008: Reject unused DEFINE variables
        I know you preferred silently ignoring unused DEFINE
        variables [1], and I agree the standard doesn't mandate
        an error. However, if we later add qualified column
        references (e.g. B AS A.price > 100), B's expression
        depends on A being present. If A is not used in PATTERN
        and the planner silently removes it, B's reference to A
        becomes dangling. I worry that silently allowing this now
        could create forward-compatibility problems once qualified
        references are introduced. For that reason, I'm inclined
        to think rejecting them now may be safer than changing
        behavior later, which would be a user-visible compatibility
        break. This is also consistent with Oracle's behavior
        (ORA-62503), as SungJun reported, and it helps catch user
        typos in DEFINE variable names at parse time.

  0009: Clarify RPR documentation in advanced.sgml
        Improve absorption explanation and clarify non-match row
        aggregation behavior with concrete examples.

  0010: Fix typos in RPR comments and parser README

  0011: Clarify excludeLocation and empty quantifier in gram.y
        Add comments explaining the conditional location assignment
        pattern and the empty quantifier rule.

  0012: Clarify RPR_VARID_MAX definition
        Document varId range 0-250 and reserved control element
        values 252+.

  0013: Move local variables to function scope
        In row_is_in_reduced_frame, move declarations out of
        switch case blocks.

  0014: Reset reduced_frame_map pointer in release_partition
        Set reduced_frame_map = NULL and alloc_sz = 0 to prevent
        dangling pointer after partition context reset.

  0015: Remove redundant list manipulation in nfa_add_matched_state
        Simplify doubly-linked list operation that was duplicated
        by the subsequent ExecRPRFreeContext() call.

  experimental: Implement 1-slot PREV/NEXT navigation
        The slot-based PREV/NEXT approach I mentioned earlier.
        This no longer needs attno_map, so the plan cache mutation
        issue is gone. PREV/NEXT now also accepts an optional
        second argument for offset, e.g. PREV(price, 2).

        Note: all existing regression tests pass with this patch,
        but my understanding of TupleSlot internals and JIT is still
        shallow -- the implementation largely copies surrounding
        patterns, so there may be issues I haven't caught yet. I'd
        like to keep this as a separate patch on top of v46 until it
        is trustworthy enough, rather than folding it into the main
        series. I'll continue reviewing and hardening it.

One design decision regarding PREV/NEXT: the SQL standard
requires that the first argument of PREV/NEXT contain at least
one column reference [2] -- e.g. PREV(1) is a syntax error.
Beyond the standard's rationale (no starting row for offsetting),
there is a practical reason: when the target row falls outside
the frame, PREV should return NULL, but a constant expression
like PREV(42) would still evaluate to 42 since it never reads
from the slot. I think it's correct to follow the standard and
reject PREV/NEXT without a column reference. What do you think?

[1]
https://www.postgresql.org/message-id/20260305.142049.1864331791480656300.ishii%40postgresql.org
[2] ISO/IEC 19075-5, Subclause 5.6.2

Best regards,
Henson

  286. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-21T05:02:32Z

    Hi Henson,

> Hi Tatsuo,
> 
> Sorry for sending these before your next revision of v45 --
> I'll be busy with a project proposal for the next couple of
> weeks, so I wanted to share them before that.

No problem at all. Thank you for your work.

> Here are the patches on top of v45, incorporating the fixes and
> improvements I mentioned earlier, plus additional changes from a
> thorough code review. There are quite a few patches since I went
> ahead without prior discussion -- please review each one individually
> and feel free to drop any that you disagree with.
> 
> Here's a summary of each patch (to be applied on top of v45):
> 
>   0001: Fix mergeGroupPrefixSuffix() max increment
>         When absorbing prefix/suffix, child->max was not incremented,
>         causing incorrect quantifier bounds. (reported by Zsolt)

Looks good to me.

>   0002: Fix RPR error codes and GROUPS typo
>         Use ERRCODE_WINDOWING_ERROR instead of ERRCODE_SYNTAX_ERROR
>         for semantic window-frame violations. Fix "GROUP" typo to
>         "GROUPS" in frame validation error message. (reported by Zsolt)

Looks good to me.

>   0003: Add check_stack_depth() and CHECK_FOR_INTERRUPTS()
>         Add stack depth protection to recursive pattern optimization
>         and interrupt checks to NFA engine loops.

Looks good to me.

>   0004: Fix window_last_value set_mark during RPR
>         Restore set_mark=true in window_last_value (normal behavior)
>         and add an RPR-specific override inside WinGetFuncArgInFrame.
>         This keeps the RPR workaround localized rather than changing
>         the caller's semantics unconditionally.

So, if RPR is active, WinSetMarkPosition() is not called at all?  That
seems too strong limitation.  Can't we set mark at frameheadpos even
if RPR is active? It seems safe since the only allowed franme start
option is ROWS BETWEEN CURRENT ROW and we never step back to the rows
before frameheadpos.

>   0005: Fix row_is_in_reduced_frame in WINDOW_SEEK_TAIL
>         Pass frameheadpos directly to row_is_in_reduced_frame
>         instead of frameheadpos+relpos. Currently only last_value()
>         calls this path with relpos=0 so no actual bug, but the
>         old expression would be incorrect for negative relpos.
>         Also add a bounds check for future callers.

Looks good to me.

>   0006: Clarify ST_NONE intent
>         Add comment explaining ST_NONE = 0 is the default for
>         non-RPR windows.

Looks good to me.

>   0007: Clarify inverse transition optimization comment
>         Document why RPR disables inverse transition: the reduced
>         frame changes row by row.

Looks good to me.

>   0008: Reject unused DEFINE variables
>         I know you preferred silently ignoring unused DEFINE
>         variables [1], and I agree the standard doesn't mandate
>         an error. However, if we later add qualified column
>         references (e.g. B AS A.price > 100), B's expression
>         depends on A being present. If A is not used in PATTERN
>         and the planner silently removes it, B's reference to A
>         becomes dangling. I worry that silently allowing this now
>         could create forward-compatibility problems once qualified
>         references are introduced. For that reason, I'm inclined
>         to think rejecting them now may be safer than changing
>         behavior later, which would be a user-visible compatibility
>         break. This is also consistent with Oracle's behavior
>         (ORA-62503), as SungJun reported, and it helps catch user
>         typos in DEFINE variable names at parse time.

Ok, let's throw an error in the case then.

>   0009: Clarify RPR documentation in advanced.sgml
>         Improve absorption explanation and clarify non-match row
>         aggregation behavior with concrete examples.

Looks good to me.

>   0010: Fix typos in RPR comments and parser README

Looks good to me.

>   0011: Clarify excludeLocation and empty quantifier in gram.y
>         Add comments explaining the conditional location assignment
>         pattern and the empty quantifier rule.

Looks good to me.

>   0012: Clarify RPR_VARID_MAX definition
>         Document varId range 0-250 and reserved control element
>         values 252+.

Looks good to me.

>   0013: Move local variables to function scope
>         In row_is_in_reduced_frame, move declarations out of
>         switch case blocks.

Looks good to me.

>   0014: Reset reduced_frame_map pointer in release_partition
>         Set reduced_frame_map = NULL and alloc_sz = 0 to prevent
>         dangling pointer after partition context reset.

Looks good to me.

>   0015: Remove redundant list manipulation in nfa_add_matched_state
>         Simplify doubly-linked list operation that was duplicated
>         by the subsequent ExecRPRFreeContext() call.

Looks good to me.

>   experimental: Implement 1-slot PREV/NEXT navigation

Great work!

>         The slot-based PREV/NEXT approach I mentioned earlier.
>         This no longer needs attno_map, so the plan cache mutation
>         issue is gone. PREV/NEXT now also accepts an optional
>         second argument for offset, e.g. PREV(price, 2).
> 
>         Note: all existing regression tests pass with this patch,
>         but my understanding of TupleSlot internals and JIT is still
>         shallow -- the implementation largely copies surrounding
>         patterns, so there may be issues I haven't caught yet. I'd
>         like to keep this as a separate patch on top of v46 until it
>         is trustworthy enough, rather than folding it into the main
>         series. I'll continue reviewing and hardening it.

Agreed. Let's keep this as a separate patch.

> One design decision regarding PREV/NEXT: the SQL standard
> requires that the first argument of PREV/NEXT contain at least
> one column reference [2] -- e.g. PREV(1) is a syntax error.
> Beyond the standard's rationale (no starting row for offsetting),
> there is a practical reason: when the target row falls outside
> the frame, PREV should return NULL, but a constant expression
> like PREV(42) would still evaluate to 42 since it never reads
> from the slot. I think it's correct to follow the standard and
> reject PREV/NEXT without a column reference. What do you think?

I agree with you. Let's follow the standard/

I am going to work on v46 patch sets.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  287. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-03-21T12:42:52Z

    Hi Tatsuo,

>   0004: Fix window_last_value set_mark during RPR
> >         Restore set_mark=true in window_last_value (normal behavior)
> >         and add an RPR-specific override inside WinGetFuncArgInFrame.
> >         This keeps the RPR workaround localized rather than changing
> >         the caller's semantics unconditionally.
>
> So, if RPR is active, WinSetMarkPosition() is not called at all?  That
> seems too strong limitation.  Can't we set mark at frameheadpos even
> if RPR is active? It seems safe since the only allowed franme start
> option is ROWS BETWEEN CURRENT ROW and we never step back to the rows
> before frameheadpos.


You raise a valid point. Setting mark at frameheadpos would be safe
for the window function's own read pointer, since we only allow
ROWS BETWEEN CURRENT ROW.

However, the difficulty is with the PREV/NEXT navigation. In the
experimental patch, PREV accepts an optional offset argument --
e.g. PREV(price, N) -- and this offset can be a runtime expression
whose value is not known until evaluation time. If the mark is
advanced to frameheadpos, PREV(price, N) with a large N could try
to access rows that the tuplestore has already truncated.

This is why the experimental patch takes a different approach: it
creates a separate nav_winobj with its own mark pointer pinned at
position 0, so that the tuplestore never truncates rows that PREV
might need. The window function's own mark management is left
unchanged.

The mark=0 approach is safe but conservative -- it prevents the
tuplestore from releasing any rows within the partition. The SQL
standard (ISO/IEC 19075-5, Subclause 5.6.2) requires the offset
argument to be a "runtime constant" -- meaning it cannot reference
columns or row pattern variables. So in practice, the maximum
offset is always known at plan time, which opens the door for a
future optimization: advance the mark to (currentpos - max_offset)
when all PREV offsets in the DEFINE clause are constant.

Since the experimental patch redesigns how mark works for PREV/NEXT
navigation, I think we should drop 0004 from the current series
and revisit the set_mark question together with the experimental
patch. What do you think?

Best regards,
Henson

  288. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-21T14:16:29Z

    Hi Henson,

>> So, if RPR is active, WinSetMarkPosition() is not called at all?  That
>> seems too strong limitation.  Can't we set mark at frameheadpos even
>> if RPR is active? It seems safe since the only allowed franme start
>> option is ROWS BETWEEN CURRENT ROW and we never step back to the rows
>> before frameheadpos.
> 
> 
> You raise a valid point. Setting mark at frameheadpos would be safe
> for the window function's own read pointer, since we only allow
> ROWS BETWEEN CURRENT ROW.
> 
> However, the difficulty is with the PREV/NEXT navigation. In the
> experimental patch, PREV accepts an optional offset argument --
> e.g. PREV(price, N) -- and this offset can be a runtime expression
> whose value is not known until evaluation time. If the mark is
> advanced to frameheadpos, PREV(price, N) with a large N could try
> to access rows that the tuplestore has already truncated.
>
> This is why the experimental patch takes a different approach: it
> creates a separate nav_winobj with its own mark pointer pinned at
> position 0, so that the tuplestore never truncates rows that PREV
> might need. The window function's own mark management is left
> unchanged.
>
> The mark=0 approach is safe but conservative -- it prevents the
> tuplestore from releasing any rows within the partition. The SQL
> standard (ISO/IEC 19075-5, Subclause 5.6.2) requires the offset
> argument to be a "runtime constant" -- meaning it cannot reference
> columns or row pattern variables. So in practice, the maximum
> offset is always known at plan time, which opens the door for a
> future optimization: advance the mark to (currentpos - max_offset)
> when all PREV offsets in the DEFINE clause are constant.
>
> Since the experimental patch redesigns how mark works for PREV/NEXT
> navigation, I think we should drop 0004 from the current series
> and revisit the set_mark question together with the experimental
> patch. What do you think?

Agreed. So I just want to make sure I create v46 using 0001-0003 and
0005-0015. Am I correct?

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  289. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-03-21T14:57:06Z

    2026년 3월 21일 (토) 23:16, Tatsuo Ishii <ishii@postgresql.org>님이 작성:

> Hi Henson,
>
> >> So, if RPR is active, WinSetMarkPosition() is not called at all?  That
> >> seems too strong limitation.  Can't we set mark at frameheadpos even
> >> if RPR is active? It seems safe since the only allowed franme start
> >> option is ROWS BETWEEN CURRENT ROW and we never step back to the rows
> >> before frameheadpos.
> >
> >
> > You raise a valid point. Setting mark at frameheadpos would be safe
> > for the window function's own read pointer, since we only allow
> > ROWS BETWEEN CURRENT ROW.
> >
> > However, the difficulty is with the PREV/NEXT navigation. In the
> > experimental patch, PREV accepts an optional offset argument --
> > e.g. PREV(price, N) -- and this offset can be a runtime expression
> > whose value is not known until evaluation time. If the mark is
> > advanced to frameheadpos, PREV(price, N) with a large N could try
> > to access rows that the tuplestore has already truncated.
> >
> > This is why the experimental patch takes a different approach: it
> > creates a separate nav_winobj with its own mark pointer pinned at
> > position 0, so that the tuplestore never truncates rows that PREV
> > might need. The window function's own mark management is left
> > unchanged.
> >
> > The mark=0 approach is safe but conservative -- it prevents the
> > tuplestore from releasing any rows within the partition. The SQL
> > standard (ISO/IEC 19075-5, Subclause 5.6.2) requires the offset
> > argument to be a "runtime constant" -- meaning it cannot reference
> > columns or row pattern variables. So in practice, the maximum
> > offset is always known at plan time, which opens the door for a
> > future optimization: advance the mark to (currentpos - max_offset)
> > when all PREV offsets in the DEFINE clause are constant.
> >
> > Since the experimental patch redesigns how mark works for PREV/NEXT
> > navigation, I think we should drop 0004 from the current series
> > and revisit the set_mark question together with the experimental
> > patch. What do you think?
>
> Agreed. So I just want to make sure I create v46 using 0001-0003 and
> 0005-0015. Am I correct?


Yes, that’s right!​​​​​​​​​​​​​​​​


> Best regards,
> --
> Tatsuo Ishii
> SRA OSS K.K.
> English: http://www.sraoss.co.jp/index_en/
> Japanese:http://www.sraoss.co.jp
>

  290. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-22T05:23:26Z

    >> Agreed. So I just want to make sure I create v46 using 0001-0003 and
>> 0005-0015. Am I correct?
> 
> 
> Yes, that’s right!​​​​​​​​​​​​​​​​

Ok, attached is the v46 patches. Here some comments:

- As git apply did not like 0003, I applied it using patch command.

- 0010 did not apply because 0004 skipped. I modified 0010 a little
  bit and applied it.

- Henson's "experimental" implementation of PREV/NEXT are not included
  in the patch set.

Here are differences from v45.

- Number of patch files are now 9, not 8. As we discussed, I split
  the regression test patches into two separate patch files: SQL +
  data patches and expected files. It seems CFBot does not work
  smoothly if a patch file is too large. We hope the splitting makes
  CFBot work more smoothly.

Below are comments from Henson. See
https://www.postgresql.org/message-id/CAAAe_zBbrnx2fjK2s%2BJgx6TSOdnKAPawXbHeX49WqmX9ji%2BHdg%40mail.gmail.com
for more details.

 0001: Fix mergeGroupPrefixSuffix() max increment
        When absorbing prefix/suffix, child->max was not incremented,
        causing incorrect quantifier bounds. (reported by Zsolt)

  0002: Fix RPR error codes and GROUPS typo
        Use ERRCODE_WINDOWING_ERROR instead of ERRCODE_SYNTAX_ERROR
        for semantic window-frame violations. Fix "GROUP" typo to
        "GROUPS" in frame validation error message. (reported by Zsolt)

  0003: Add check_stack_depth() and CHECK_FOR_INTERRUPTS()
        Add stack depth protection to recursive pattern optimization
        and interrupt checks to NFA engine loops.

  0005: Fix row_is_in_reduced_frame in WINDOW_SEEK_TAIL
        Pass frameheadpos directly to row_is_in_reduced_frame
        instead of frameheadpos+relpos. Currently only last_value()
        calls this path with relpos=0 so no actual bug, but the
        old expression would be incorrect for negative relpos.
        Also add a bounds check for future callers.

  0006: Clarify ST_NONE intent
        Add comment explaining ST_NONE = 0 is the default for
        non-RPR windows.

  0007: Clarify inverse transition optimization comment
        Document why RPR disables inverse transition: the reduced
        frame changes row by row.

  0008: Reject unused DEFINE variables
        I know you preferred silently ignoring unused DEFINE
        variables [1], and I agree the standard doesn't mandate
        an error. However, if we later add qualified column
        references (e.g. B AS A.price > 100), B's expression
        depends on A being present. If A is not used in PATTERN
        and the planner silently removes it, B's reference to A
        becomes dangling. I worry that silently allowing this now
        could create forward-compatibility problems once qualified
        references are introduced. For that reason, I'm inclined
        to think rejecting them now may be safer than changing
        behavior later, which would be a user-visible compatibility
        break. This is also consistent with Oracle's behavior
        (ORA-62503), as SungJun reported, and it helps catch user
        typos in DEFINE variable names at parse time.

  0009: Clarify RPR documentation in advanced.sgml
        Improve absorption explanation and clarify non-match row
        aggregation behavior with concrete examples.

  0010: Fix typos in RPR comments and parser README

  0011: Clarify excludeLocation and empty quantifier in gram.y
        Add comments explaining the conditional location assignment
        pattern and the empty quantifier rule.

  0012: Clarify RPR_VARID_MAX definition
        Document varId range 0-250 and reserved control element
        values 252+.

  0013: Move local variables to function scope
        In row_is_in_reduced_frame, move declarations out of
        switch case blocks.

  0014: Reset reduced_frame_map pointer in release_partition
        Set reduced_frame_map = NULL and alloc_sz = 0 to prevent
        dangling pointer after partition context reset.

  0015: Remove redundant list manipulation in nfa_add_matched_state
        Simplify doubly-linked list operation that was duplicated
        by the subsequent ExecRPRFreeContext() call.

Best regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  291. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-03-29T12:13:47Z

    Hi Tatsuo,

Attached are 6 incremental patches on top of v46.

0001-0005 are small independent fixes. 0006 is the main
PREV/NEXT navigation redesign that was discussed as the
"experimental" implementation in prior messages.

Here is a summary of each patch:


  0001: Remove unused regex/regex.h include from nodeWindowAgg.c
        The regex header was left over from an earlier implementation
        that used the regex engine for pattern matching. The current
        NFA engine in execRPR.c does not use it.


  0002: Add CHECK_FOR_INTERRUPTS() to nfa_add_state_unique()
        This function iterates through a linked list of NFA states
        to check for duplicates. In state explosion scenarios
        (complex patterns with alternations and quantifiers), this
        list can grow significantly. Without an interrupt check,
        the loop is unresponsive to query cancellation.


  0003: Add CHECK_FOR_INTERRUPTS() to nfa_try_absorb_context()
        Similar to 0002. The absorption loop iterates through the
        context chain, which can grow unbounded with streaming
        window patterns. Each iteration also calls
        nfa_states_covered() which has its own loop.


  0004: Fix in-place modification of defineClause TargetEntry
        In set_upper_references(), the defineClause TargetEntry
        was modified in-place by fix_upper_expr() without making
        a copy first. This follows the same flatCopyTargetEntry()
        pattern already used for the main targetlist in the same
        function.


  0005: Fix mark handling for last_value() under RPR
        This is a revised version of the v45 0004 patch that you
        commented on [1]. You pointed out that suppressing
        WinSetMarkPosition() entirely under RPR was too strong a
        limitation, and we agreed to drop it and revisit together
        with the experimental PREV/NEXT patch.

        The RPR executor patch changed set_mark from true to
        false in window_last_value() to avoid mark advancement
        problems under RPR. But this also penalizes non-RPR
        queries by preventing tuplestore memory reclamation.

        The revised approach: restore set_mark=true (upstream
        default), and add a targeted guard in
        WinGetFuncArgInFrame() that only suppresses mark
        advancement for SEEK_TAIL under RPR -- not for all
        seek types as in v45 0004. Only SEEK_TAIL needs
        suppression because advancing the mark from the tail
        could prevent revisiting earlier rows that still fall
        within a future row's reduced frame.

        [1]
https://www.postgresql.org/message-id/20260321.140232.1947157589839395257.ishii%40postgresql.org


  0006: Implement 1-slot PREV/NEXT navigation for RPR
        This is the main patch. It redesigns PREV/NEXT navigation
        from the 3-slot model (outer/scan/inner with varno
        rewriting) to a 1-slot model using expression opcodes.

        Key changes:

        - RPRNavExpr: a new expression node type that replaces
          the previous approach of identifying PREV/NEXT by
          funcid. The parser transforms PREV/NEXT function calls
          into RPRNavExpr nodes in ParseFuncOrColumn().

        - EEOP_RPR_NAV_SET/RESTORE: two new opcodes that
          temporarily swap ecxt_outertuple to the target row
          during expression evaluation. The argument expression
          evaluates against the swapped slot, then the original
          slot is restored. This eliminates varno rewriting and
          naturally supports arbitrary offsets.

        - 2-argument form: PREV(value, offset) and
          NEXT(value, offset) are now supported. The offset
          defaults to 1 if omitted; offset=0 refers to the
          current row. The offset must be a run-time constant
          per the SQL standard.

        - nav_winobj: a dedicated WindowObject with its own
          tuplestore read pointer, separate from aggregate
          processing. A mark pointer pinned at position 0
          prevents tuplestore truncation so that PREV(expr, N)
          can reach any prior row. This is conservative -- it
          retains the entire partition -- but since the standard
          requires offsets to be run-time constants, we could
          advance the mark to (currentpos - max_offset) in a
          future optimization. I'd prefer to defer that until
          the basic navigation is stable.

        - Documentation and tests updated.


Changes from the experimental version of 0006:

- NULL/negative offset error rationale changed from "matching
  Oracle behavior" to the SQL standard (ISO/IEC 9075-2,
  Subclause 5.6.2: "There is an exception if the value of
  the offset is negative or null"). The behavior is the same;
  only the comment was corrected.

- RPRNavKind changed from -1/+1 values to plain enum values.
  The previous version used kind as an arithmetic multiplier
  (offset * kind), but this pattern cannot extend to FIRST/LAST
  which have different position semantics. The new version uses
  a switch statement with pg_sub/add_s64_overflow instead.

- Parser validation walkers consolidated from three separate
  walkers into a single NavCheckResult struct + nav_check_walker,
  reducing tree traversals from up to 4 per RPRNavExpr to 1-2.

- JIT changed from attempting to compile to explicit interpreter
  fallback. See item 2 below for the rationale.

- Additional test cases for subquery offset (caught by
  DEFINE-level restriction), first-arg subquery, and first-arg
  volatile function (allowed per standard).

I've reviewed and revised 0006 since the experimental version.
I think it is now ready to be included in the next version of
the patch set. It passes all existing regression tests (rpr,
rpr_base, rpr_explain, rpr_nfa) and adds comprehensive tests
for the new functionality.

Two items I'd like your opinion on:

1. 0005 (mark handling): The revised approach only suppresses
   mark advancement for SEEK_TAIL under RPR, unlike v45 0004
   which suppressed it for all seek types. Does narrowing to
   SEEK_TAIL seem reasonable to you, or do you see cases where
   other seek types could also be affected?

2. LLVM JIT fallback (in 0006): The mid-expression slot swap
   conflicts with how JIT caches tuple data pointers. The
   experimental version produced wrong results under
   jit_above_cost=0, and I have not found a clean fix within
   the current JIT framework. For now, DEFINE expressions
   containing PREV/NEXT fall back to the interpreter; other
   expressions in the same query are still JIT-compiled.
   I'd like to keep this as-is for now and look for a proper
   JIT solution over time. Does that sound reasonable?


Best regards,
Henson

  292. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-03-29T14:06:11Z

    Hi Tatsuo,

Separately from the PREV/NEXT patches, I have been analyzing
how FIRST and LAST navigation would interact with the current
NFA engine -- particularly context absorption and DEFINE
evaluation sharing. This is not a proposal to implement them
now; rather, I wanted to document the design constraints
while the analysis is fresh, so we have a reference when we
are ready to add them.


Design Note: FIRST/LAST Navigation for RPR
===========================================

Scope: This note covers only patterns without quantifiers, where
each pattern variable consumes exactly one row.


1. Navigation Function Classification
--------------------------------------

The SQL standard (ISO/IEC 19075-5:2021) defines four navigation
functions in two categories:

PREV/NEXT -- physical navigation:
  - Scope: entire row pattern partition
  - Offset: physical (row position within partition)
  - Base row: the row denoted by the variable under running
    semantics (in our no-qualifier scope: current eval row)
  - Default offset: 1
  - Can reach rows outside the match (standard 5.6.2)
  - Always RUNNING semantics only

FIRST/LAST -- logical navigation:
  - Standard scope: rows mapped to a specific variable
  - Our scope (no quantifier): match_start to currentpos range
  - Offset: logical (in standard), but equivalent to physical
    when each variable maps to at most one row
  - Base: match_start (FIRST) or currentpos (LAST)
  - Default offset: 0 (unlike PREV/NEXT which default to 1)
  - RUNNING semantics in DEFINE; FINAL available in MEASURES

FIRST/LAST results depend on the match start position
(match_start). Each current row in window RPR triggers an
independent match attempt with its own match_start. FIRST
always depends on match_start. LAST(price) depends only on
currentpos and is match_start-independent, but LAST(price, N)
depends on match_start because the NULL boundary check
(currentpos - N < match_start) varies with match_start.

Example (PATTERN (A B C), match_start = 3):

    FIRST(price)    = R3.price   (match_start row)
    FIRST(price, 1) = R4.price   (match_start + 1)
    LAST(price)     = R5.price   (currentpos row)
    LAST(price, 1)  = R4.price   (currentpos - 1)
    LAST(price, 3)  = NULL       (before match_start)

    PREV(price, 2)  = R3.price   (pos - 2, physical)
    NEXT(price, 2)  = R7.price   (pos + 2, beyond match)

PREV/NEXT can reach any row in the partition regardless of
match boundaries. FIRST/LAST are valid only within the
match_start to currentpos range.


2. match_start and NFA Contexts
-------------------------------

Each match_start defines one match attempt, which corresponds
to one NFA context. Different contexts have different
match_start values. Context absorption merges contexts when
one context's pattern states are covered by another's. If the
merged contexts have different match_start values, FIRST/LAST
would produce incorrect results because the surviving context's
match_start would be used for both.


3. Compound Navigation (PREV/NEXT wrapping FIRST/LAST)
------------------------------------------------------

The standard (5.6.4) permits nesting FIRST/LAST inside
PREV/NEXT. Using the standard's notation with variable
qualifiers (which our no-quantifier scope does not yet
require):

    PREV(LAST(A.Price + A.Tax, 1), 3)

Evaluation proceeds in three steps:
  1. Inner FIRST/LAST: determines target row position only
     (does not evaluate the expression)
  2. Outer PREV/NEXT: applies physical offset from that row
  3. Expression evaluation: at the final destination row only

The reverse nesting (FIRST/LAST wrapping PREV/NEXT) and
same-kind nesting (PREV inside PREV, etc.) are prohibited.

Since the inner function only determines a position, the
compound operation can be flattened into a single RPRNavExpr
node internally, requiring only one slot swap.


4. Context Absorption Rules
----------------------------

Absorption safety depends on whether the navigation function
depends on match_start:

  DEFINE content                  | match_start dep. | absorption
  --------------------------------|------------------|----------
  No navigation                   | none             | safe
  PREV/NEXT only                  | none             | safe
  LAST (no offset)                | none             | safe
  LAST (with offset)              | boundary check   | unsafe
  FIRST (any)                     | direct           | unsafe
  Compound (inner FIRST)          | direct           | unsafe
  Compound (inner LAST, no off.)  | none             | safe
  Compound (inner LAST, with off.)| boundary check   | unsafe

The criterion: if the expression depends on match_start in
any way, absorption is unsafe. Comparing match_start values
at runtime is impractical, so we treat any dependency as
unconditionally unsafe.


5. DEFINE Clause Evaluation Rules
---------------------------------

Currently, nfa_evaluate_row(pos) evaluates all DEFINE
variables once per row and shares the results across all NFA
contexts.

  DEFINE content                  | evaluation
  --------------------------------|--------------------------
  No navigation                   | shared (once per row)
  PREV/NEXT only                  | shared (once per row)
  LAST (no offset)                | shared (once per row)
  LAST (with offset)              | per-context
  FIRST (any)                     | per-context
  Compound (inner FIRST)          | per-context
  Compound (inner LAST, no off.)  | shared (once per row)
  Compound (inner LAST, with off.)| per-context


6. Slot Swap Elision
--------------------

When the final target position resolves to currentpos, the
slot swap can be elided. This applies to:
  - PREV(price, 0), NEXT(price, 0): offset 0
  - LAST(price), LAST(price, 0): always currentpos
  - Compound cases where offsets cancel, e.g.,
    NEXT(LAST(price, 1), 1)


The PREV/NEXT infrastructure (RPRNavExpr, EEOP_RPR_NAV_SET/
RESTORE, nav_winobj) can be reused directly for FIRST/LAST --
it would mainly require adding two RPRNavKind values and
match_start arithmetic in the ExecEvalRPRNavSet switch.
No variable qualifiers, match history tracking, or CLASSIFIER
infrastructure would be needed for this no-quantifier scope.

I do think CLASSIFIER, MEASURES, and match history tracking
should stay out of the initial patch set -- they add
significant complexity to NFA contexts and affect absorption
in ways that are hard to validate incrementally. Variable
qualifiers and quantifier support for navigation functions
are also tied to match history infrastructure, so they should
be deferred as well. Better to get the current feature set
stable first.

With PREV/NEXT now resolved (0006), the next major milestone
would be match history infrastructure (needed for CLASSIFIER,
MEASURES, variable qualifiers, and quantifier-aware
navigation). Before that larger step, there are a few
remaining items that can be done within the current
architecture:

Optimization:
  - PREFIX pattern absorption (e.g., A (B+) C) [1][2]
  - Fixed-length iteration group absorption
    (e.g., (A B{2})+ ) [3]

Feature:
  - FIRST/LAST navigation (this note)
  - LLVM JIT proper support for slot swap (0006 uses
    interpreter fallback) [4][5]

[1]
https://www.postgresql.org/message-id/CAAAe_zAEg7sVM%3DWDwXMyE-odGmQyXSVi5ZzWgye6SupSjdMKpg%40mail.gmail.com
[2]
https://www.postgresql.org/message-id/CAAAe_zDq7R8CaDfmh8C%2BH3_639Y5LtJD%2B2Z%3D1txDt%3DvaOr90rQ%40mail.gmail.com
[3]
https://www.postgresql.org/message-id/CAAAe_zBY%2BEqk_DQpS0cy1Eb67H9v92tyaf%2BU%2BSKKuLGUs_z%2BEA%40mail.gmail.com
[4]
https://www.postgresql.org/message-id/CAAAe_zBbrnx2fjK2s%2BJgx6TSOdnKAPawXbHeX49WqmX9ji%2BHdg%40mail.gmail.com
[5]
https://www.postgresql.org/message-id/CAAAe_zBCF3dwSjStmG0kJqw_y1z8QD73Rf1G58QTKEvd9tScwA%40mail.gmail.com

The question is how much of this to complete before the
current patches are committed. The patch set is already
substantial, and the next CommitFest is in July. What do
you think -- should we try to include some of these items
in the current round, or focus on getting what we have
committed first? Also, do you see any blockers that might
prevent the current patch set from being committed?


Best regards,
Henson

>

  293. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-03-30T04:34:28Z

    Hi Henson,

> Hi Tatsuo,
> 
> Attached are 6 incremental patches on top of v46.

Thank you!

> 0001-0005 are small independent fixes. 0006 is the main
> PREV/NEXT navigation redesign that was discussed as the
> "experimental" implementation in prior messages.
> 
> Here is a summary of each patch:
> 
> 
>   0001: Remove unused regex/regex.h include from nodeWindowAgg.c
>         The regex header was left over from an earlier implementation
>         that used the regex engine for pattern matching. The current
>         NFA engine in execRPR.c does not use it.

Good catch!

>   0002: Add CHECK_FOR_INTERRUPTS() to nfa_add_state_unique()
>         This function iterates through a linked list of NFA states
>         to check for duplicates. In state explosion scenarios
>         (complex patterns with alternations and quantifiers), this
>         list can grow significantly. Without an interrupt check,
>         the loop is unresponsive to query cancellation.

Looks good to me.

>   0003: Add CHECK_FOR_INTERRUPTS() to nfa_try_absorb_context()
>         Similar to 0002. The absorption loop iterates through the
>         context chain, which can grow unbounded with streaming
>         window patterns. Each iteration also calls
>         nfa_states_covered() which has its own loop.

Looks good to me.

>   0004: Fix in-place modification of defineClause TargetEntry
>         In set_upper_references(), the defineClause TargetEntry
>         was modified in-place by fix_upper_expr() without making
>         a copy first. This follows the same flatCopyTargetEntry()
>         pattern already used for the main targetlist in the same
>         function.

Probably we want to modify the comment above since it implies an
in-place modification? What about something like this? (Modifies -> Replace)

	/*
	 * Replace an expression tree in each DEFINE clause so that all Var
	 * nodes's varno refers to OUTER_VAR.
	 */

>   0005: Fix mark handling for last_value() under RPR
>         This is a revised version of the v45 0004 patch that you
>         commented on [1]. You pointed out that suppressing
>         WinSetMarkPosition() entirely under RPR was too strong a
>         limitation, and we agreed to drop it and revisit together
>         with the experimental PREV/NEXT patch.
> 
>         The RPR executor patch changed set_mark from true to
>         false in window_last_value() to avoid mark advancement
>         problems under RPR. But this also penalizes non-RPR
>         queries by preventing tuplestore memory reclamation.
> 
>         The revised approach: restore set_mark=true (upstream
>         default), and add a targeted guard in
>         WinGetFuncArgInFrame() that only suppresses mark
>         advancement for SEEK_TAIL under RPR -- not for all
>         seek types as in v45 0004. Only SEEK_TAIL needs
>         suppression because advancing the mark from the tail
>         could prevent revisiting earlier rows that still fall
>         within a future row's reduced frame.

I think instead we can set a mark at frameheadpos when seek type is
SEEK_TAIL and RPR is enabled. See attached patch.

>   0006: Implement 1-slot PREV/NEXT navigation for RPR
>         This is the main patch. It redesigns PREV/NEXT navigation
>         from the 3-slot model (outer/scan/inner with varno
>         rewriting) to a 1-slot model using expression opcodes.
> 
>         Key changes:
> 
>         - RPRNavExpr: a new expression node type that replaces
>           the previous approach of identifying PREV/NEXT by
>           funcid. The parser transforms PREV/NEXT function calls
>           into RPRNavExpr nodes in ParseFuncOrColumn().
> 
>         - EEOP_RPR_NAV_SET/RESTORE: two new opcodes that
>           temporarily swap ecxt_outertuple to the target row
>           during expression evaluation. The argument expression
>           evaluates against the swapped slot, then the original
>           slot is restored. This eliminates varno rewriting and
>           naturally supports arbitrary offsets.
> 
>         - 2-argument form: PREV(value, offset) and
>           NEXT(value, offset) are now supported. The offset
>           defaults to 1 if omitted; offset=0 refers to the
>           current row. The offset must be a run-time constant
>           per the SQL standard.
> 
>         - nav_winobj: a dedicated WindowObject with its own
>           tuplestore read pointer, separate from aggregate
>           processing. A mark pointer pinned at position 0
>           prevents tuplestore truncation so that PREV(expr, N)
>           can reach any prior row. This is conservative -- it
>           retains the entire partition -- but since the standard
>           requires offsets to be run-time constants, we could
>           advance the mark to (currentpos - max_offset) in a
>           future optimization. I'd prefer to defer that until
>           the basic navigation is stable.
> 
>         - Documentation and tests updated.
>
>
> Changes from the experimental version of 0006:
> 
> - NULL/negative offset error rationale changed from "matching
>   Oracle behavior" to the SQL standard (ISO/IEC 9075-2,
>   Subclause 5.6.2: "There is an exception if the value of
>   the offset is negative or null"). The behavior is the same;
>   only the comment was corrected.
> 
> - RPRNavKind changed from -1/+1 values to plain enum values.
>   The previous version used kind as an arithmetic multiplier
>   (offset * kind), but this pattern cannot extend to FIRST/LAST
>   which have different position semantics. The new version uses
>   a switch statement with pg_sub/add_s64_overflow instead.
> 
> - Parser validation walkers consolidated from three separate
>   walkers into a single NavCheckResult struct + nav_check_walker,
>   reducing tree traversals from up to 4 per RPRNavExpr to 1-2.
> 
> - JIT changed from attempting to compile to explicit interpreter
>   fallback. See item 2 below for the rationale.
> 
> - Additional test cases for subquery offset (caught by
>   DEFINE-level restriction), first-arg subquery, and first-arg
>   volatile function (allowed per standard).
> 
> I've reviewed and revised 0006 since the experimental version.
> I think it is now ready to be included in the next version of
> the patch set. It passes all existing regression tests (rpr,
> rpr_base, rpr_explain, rpr_nfa) and adds comprehensive tests
> for the new functionality.

Excellent! I will take a look at it. (it will take for a while).

> Two items I'd like your opinion on:
> 
> 1. 0005 (mark handling): The revised approach only suppresses
>    mark advancement for SEEK_TAIL under RPR, unlike v45 0004
>    which suppressed it for all seek types. Does narrowing to
>    SEEK_TAIL seem reasonable to you, or do you see cases where
>    other seek types could also be affected?

See the comment above.

> 2. LLVM JIT fallback (in 0006): The mid-expression slot swap
>    conflicts with how JIT caches tuple data pointers. The
>    experimental version produced wrong results under
>    jit_above_cost=0, and I have not found a clean fix within
>    the current JIT framework. For now, DEFINE expressions
>    containing PREV/NEXT fall back to the interpreter; other
>    expressions in the same query are still JIT-compiled.
>    I'd like to keep this as-is for now and look for a proper
>    JIT solution over time. Does that sound reasonable?

I am not an expert of JIT, but for me, it sounds reasonable.  We can
enhance it later on.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  294. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-04-02T03:51:05Z

    Hi Tatsuo,

Thank you for the review and the attached patch for 0005.
I appreciate you taking the time to look at each patch
carefully.

Attached are 8 incremental patches on top of v46 (replacing
the previous 6-patch set). 0001-0003 are unchanged. 0004 and
0005 incorporate your feedback below. 0006 is a new planner
fix. Previous 0006 becomes 0007. 0008 is new.

>   0004: Fix in-place modification of defineClause TargetEntry
>
> Probably we want to modify the comment above since it implies an
> in-place modification? What about something like this? (Modifies ->
> Replace)
>
>         /*
>          * Replace an expression tree in each DEFINE clause so that all Var
>          * nodes's varno refers to OUTER_VAR.
>          */
>

Good point, thank you. Applied in the updated 0004.


> >   0005: Fix mark handling for last_value() under RPR
>
> I think instead we can set a mark at frameheadpos when seek type is
> SEEK_TAIL and RPR is enabled. See attached patch.
>

Thank you for the patch. Your approach is cleaner -- setting
the mark at frameheadpos is more direct than suppressing
advancement. I've adopted it in the updated 0005.


> >   0006: Implement 1-slot PREV/NEXT navigation for RPR
>
> Excellent! I will take a look at it. (it will take for a while).
>

No rush at all. In the meantime, while testing the PREV/NEXT
patch in various query patterns, I found a planner issue.
I've also added JIT support. Here is a summary of the new
patches:


  0006: Prevent removal of RPR window functions in unused
        subquery outputs

        Wrapping an RPR window query in a subquery with an
        outer aggregate causes a crash:

          SELECT count(*) FROM (
            SELECT count(*) OVER w FROM ...
            WINDOW w AS (... DEFINE A AS i > PREV(i))
          ) t;

        remove_unused_subquery_outputs() replaces unused subquery
        target entries with NULL constants. When an RPR window
        function's result is not referenced by the outer query,
        this replacement eliminates all active window functions
        for the WindowClause, causing the planner to omit the
        WindowAgg node. DEFINE clause expressions containing
        RPRNavExpr (PREV/NEXT) then lose their execution context,
        leading to a crash.

        The fix skips the NULL replacement for window functions
        whose WindowClause has a defineClause. Even when the
        window function result is unused, RPR pattern matching
        (frame reduction) must still execute -- the WindowAgg
        node must be preserved.

        An alternative would be to let the planner remove the
        window function but teach it to still generate the
        WindowAgg node when defineClause is present, even with
        no active window functions. That would be a more
        targeted optimization but requires deeper planner
        changes. Do you think the current approach is
        sufficient, or would you prefer a different strategy?


  0007: Implement 1-slot PREV/NEXT navigation for RPR
        (unchanged from previous 0006)


> > 2. LLVM JIT fallback (in 0006): The mid-expression slot swap
>
> I am not an expert of JIT, but for me, it sounds reasonable.  We can
> enhance it later on.
>

  0008: Add JIT compilation support for RPR PREV/NEXT navigation

        EEOP_OUTER_VAR normally uses slot pointers cached in the
        entry block, but the mid-expression slot swap in
        NAV_SET/RESTORE invalidates them. To avoid penalizing
        existing expressions, the reload path is only generated
        when RPR navigation opcodes are present in the
        expression (compile-time decision). Expressions without
        PREV/NEXT produce identical machine code as before.
        NAV_SET/RESTORE use the standard build_EvalXFunc pattern.

        A 100K-row PREV/NEXT test case that runs under
        jit_above_cost=0 is included in rpr.sql.

I would appreciate your thoughts on these when you have time.


Best regards,
Henson

  295. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-04-04T07:31:13Z

    Hi hackers,


I'd like to propose an extension to the context absorption
optimization for Row Pattern Recognition in the window clause:
fixed-length iteration group absorption. This builds on the
two-flag absorption design [1] and three-phase execution
framework [2], and is a sibling optimization to PREFIX pattern
absorption [3].

The existing framework requires group children to have exactly
{1,1} quantifiers. This relaxes the constraint to accept any
fixed-length quantifier (min == max), allowing patterns like
(A B{2})+ and (A (B{2} C){4})+ to benefit from absorption
with no runtime changes.

PREFIX absorption [3] handles fixed-length elements before
the unbounded repetition; this handles fixed-length elements
inside the unbounded group.


[1] Context absorption design - two-flag approach
https://www.postgresql.org/message-id/CAAAe_zCmjDRf9u19xRRbtzq%2B-7ujqadZk0izz0mkef2j%2BWpBBA%40mail.gmail.com
[2] Three-phase absorption framework (Match -> Absorb -> Advance)
https://www.postgresql.org/message-id/CAAAe_zAGLOF_qporKahnL8ajSu_eXZYsJN2M1cEeF5n5GWSNdQ%40mail.gmail.com
[3] PREFIX pattern absorption
https://www.postgresql.org/message-id/CAAAe_zDq7R8CaDfmh8C%2BH3_639Y5LtJD%2B2Z%3D1txDt%3DvaOr90rQ%40mail.gmail.com
[4] FIRST/LAST navigation design note
https://www.postgresql.org/message-id/CAAAe_zCUrKGBgZdaazdO_v9QWHsS_1DXuP%3DrLeNhO3iwsHwwbg%40mail.gmail.com


Design Note: Fixed-Length Iteration Group Absorption
=====================================================


1. Problem Definition
=====================


1.1 Current Absorption Scope

Context absorption currently requires that children of an unbounded
group have exactly {1,1} quantifiers. This means only simple patterns
like (A B)+ qualify for absorption.

Patterns like (A B{2})+ or (A (B{2} C){4})+ are excluded even though
each iteration consumes a fixed number of rows.


1.2 The Fixed-Length Constraint

The existing {1,1} check is overly restrictive. The actual requirement
for absorption is that each iteration of the group consumes a fixed
number of rows, so that the count-dominance property holds: an older
context always has a repeat count >= any newer context at the same
element.

Any group where all children have min == max (recursively) satisfies
this property, regardless of nesting depth or quantifier values.


2. Relationship to the Existing Framework
==========================================


2.1 Position in the Three-Phase Execution Model

The Match -> Absorb -> Advance model [2] is preserved as-is. This
optimization only changes compile-time eligibility analysis in the
two-flag framework [1]; the Absorb phase logic is unchanged.


2.2 Prerequisites

The same four prerequisites as existing absorption apply:

1. Greedy quantifier -- the outer group must be greedy.
2. SKIP TO PAST LAST ROW -- prior matches suppress subsequent
   start points.
3. UNBOUNDED FOLLOWING -- all contexts see the same future rows.
4. Shared DEFINE evaluation -- all contexts produce the same
   DEFINE result for a given row [4].


2.3 Extension of the Dominance Order

The existing dominance condition is unchanged:

    C_old and C_new are in the absorbable region of the same
    element, and C_old.counts >= C_new.counts.

The only change is which patterns are recognized as having an
absorbable region. The runtime count comparison works identically
regardless of the step size per iteration.


3. Correctness Argument
========================


A fixed-length quantifier (min == max) is semantically equivalent
to repeating the element that many times. Therefore any pattern
with fixed-length children can be conceptually unrolled to {1,1}
elements:

  (A B{2})+           -> (A B B)+
  (A (B{2} C){4})+    -> (A B B C B B C B B C B B C)+
  ((A{2} B{3}){2})+   -> (A A B B B A A B B B)+

The unrolled form contains only {1,1} VARs inside the unbounded
group, which is exactly the existing Case 2 that is already proven
correct for absorption.

This optimization recognizes fixed-length groups at compile time
without actually unrolling them. The runtime behavior is identical
to the unrolled form: each iteration consumes a fixed number of
rows, count-dominance holds, and absorption proceeds unchanged.


4. Compile-Time Analysis
=========================


4.1 Change to isUnboundedStart()

Currently, Case 2 in isUnboundedStart() requires all children of
an unbounded group to be simple {1,1} VARs. The relaxation:

- VAR children: accept min == max (any value)
- BEGIN children (nested subgroups): recursively verify that all
  descendants have min == max, including the subgroup's own END
  quantifier
- ALT children: reject (unchanged)

The check follows the existing depth-based traversal of the flat
element array.


4.2 Flat Array Example

Pattern: (A (B{2} C){4})+

  [0] BEGIN depth=0  min=1 max=INF   <- outer group
  [1] A     depth=1  min=1 max=1     <- 1==1 OK
  [2] BEGIN depth=1  min=4 max=4     <- subgroup
  [3] B     depth=2  min=2 max=2     <- 2==2 OK
  [4] C     depth=2  min=1 max=1     <- 1==1 OK
  [5] END   depth=1  min=4 max=4     <- 4==4 OK
  [6] END   depth=0  min=1 max=INF   <- unbounded, greedy

Verification traverses [1]-[5] confirming min==max at every
level. [6] is the outer END: unbounded and greedy, so absorption
applies.


4.3 Runtime: No Changes

The runtime absorption logic (nfa_try_absorb_context,
nfa_states_covered) and flag assignment (RPR_ELEM_ABSORBABLE,
RPR_ELEM_ABSORBABLE_BRANCH) require no changes.


5. Worked Example
==================


Pattern: (A B{2})+
Step size = 1 + 2 = 3.

Data: rows match in repeating A, B, B cycle.
Contexts that fail A at B-rows die immediately (omitted).

Row  Data  C_old             C_new            Active
r0   A     A                 -                1
r1   B     B(cnt=1)          -                1
r2   B     B(cnt=2)->END     -                1
           count=1, loops
r3   A     A                 A                2
r4   B     B(cnt=1)          B(cnt=1)         2
r5   B     B(cnt=2)->END     B(cnt=2)->END    2
           count=2            count=1
           absorb: 2 >= 1 -> C_new absorbed
           C_old loops                        1
r6   A     A                 A                2
r7   B     B(cnt=1)          B(cnt=1)         2
r8   B     B(cnt=2)->END     B(cnt=2)->END    2
           count=3            count=1
           absorb: 3 >= 1 -> C_new absorbed
           C_old loops                        1

Absorption occurs at END (the ABSORBABLE judgment point),
where both contexts synchronize after completing one full
iteration. The older context has a strictly higher group
count and absorbs the newer one.

Between END points, both contexts progress in lockstep
at the same pattern positions. Maximum concurrent contexts
during this phase = 2. Overall complexity remains O(n).


6. Non-Qualifying Patterns
===========================


Patterns where any child has min != max do NOT qualify:

  (A B+ C)+    -- B+ has min=1, max=INF -> NOT fixed
  (A B{2,5})+  -- B has min=2, max=5 -> NOT fixed
  (A B?)+      -- B? has min=0, max=1 -> NOT fixed

These patterns have variable step sizes per iteration, so
count-dominance is not guaranteed. They remain non-absorbable
under this optimization.

  296. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-04-05T12:50:53Z

    Hi Tatsuo,

Attached are 15 incremental patches on top of v46 (replacing
the previous 8-patch set). Here is a summary of the changes
from the previous version:

  0001-0005: Unchanged from the previous set.

  0006: Fix DEFINE expression handling in RPR window
        planning (revised)

        Integration tests in 0007 revealed two crashes:
        (1) subquery wrapping with outer aggregate causes
        WindowAgg removal when RPR window function output
        is unused, (2) RPR and non-RPR windows coexisting
        causes SIGSEGV from RPRNavExpr propagating to the
        wrong WindowAgg. This version extends the fix to
        extract only Var nodes via pull_var_clause()
        instead of adding the whole DEFINE expression to
        the targetlist. The allpaths.c guard is extended
        to also preserve columns referenced by DEFINE
        clauses.

  0007: Add RPR planner integration tests (new)

        Planner optimization interactions with RPR windows
        have been revealing crashes, so this is a dedicated
        test file: window dedup, subquery flattening,
        DEFINE propagation, LATERAL, recursive CTE,
        etc. Will be expanded further.

  0008: Replace reduced frame map with single match result
        (new)

        RPR processes rows sequentially and maintains at
        most one active match per start position. This
        replaces the per-row boolean frame map array with
        four scalar fields (valid, matched, start, length),
        simplifying nodeWindowAgg.c.

  0009: Add fixed-length group absorption for RPR (new)

        Extends context absorption to accept fixed-length
        quantifiers (min == max) inside unbounded groups.
        The NFA compiler optimizes (A B B)+ into
        (A B{2})+, which previously lost absorption
        eligibility because only {1,1} children
        qualified. Now (A B{2})+ and ((A{2} B{3}){2})+
        also qualify with no runtime changes. Also fixes
        two NFA bugs: (1) incorrect absorption at
        non-judgment points -- added ABSORBABLE check
        in nfa_states_covered and fixed isAbsorbable
        propagation in advance exit paths, (2) VAR
        visited bitmap moved to nfa_add_state_unique
        to prevent false cycle detection on new-iteration
        loop-back caused by generalized END chain
        traversal. Design note posted separately [1].

  0010: Rename rpr_explain test views to descriptive
        names (new)

        Replaces sequential view names (rpr_ev83, rpr_ev84)
        with descriptive names (rpr_ev_nav_prev1, etc.) for
        maintainability when adding new tests.

  0011: Implement 1-slot PREV/NEXT navigation for RPR
        (was 0007, rebased)

        Context changes only: optimizer.h include moved to
        0006, test view names updated by 0010.

  0012: Add JIT compilation support for RPR PREV/NEXT
        (was 0008, unchanged)

        Note: master recently disabled JIT by default
        [3]. A quick benchmark with RPR did not show
        measurable improvement from JIT. This patch is
        retained for correctness (the new opcodes must
        be handled when JIT is enabled), but the
        practical benefit may be limited.

  0013: Add tuplestore trim optimization for RPR PREV
        navigation (new)

        The planner extracts max_offset from all PREV
        calls in DEFINE clauses and sets mark = currentpos
        - max_offset at runtime, so only max_offset rows
        are retained. Constant expressions are folded by
        the planner; non-constant offsets (parameters,
        etc.) are evaluated by the executor at init time
        via RPR_NAV_OFFSET_NEEDS_EVAL.

  0014: Update RPR code comments to reflect 1-slot
        navigation model (new)

        Updates stale comments in execRPR.c and
        parse_rpr.c that still referenced the old 3-slot
        design.

  0015: Implement FIRST/LAST navigation for RPR
        (work in progress)

        Adds FIRST(column) and LAST(column) navigation
        functions per SQL standard. Parser, planner,
        executor, and tests are included. See the design
        note [2] for the interaction with context
        absorption.

        Not yet done: compound navigation (e.g.
        PREV(FIRST(col))), tests for FIRST/LAST offset
        variants, tuplestore mark handling for FIRST,
        and additional edge-case tests.

In the FIRST/LAST design note [2], I listed remaining
items that can be done within the current architecture.
With this patch set, all items except PREFIX pattern
absorption are now covered: fixed-length group absorption
(0009), JIT proper support (0012), and FIRST/LAST
navigation (0015). PREFIX absorption is planned
separately, but stabilizing the base patch takes
priority. If it is ready in time, would it make sense
to decide then whether to include it in this round?
Apart from bug fixes, are there any features you
consider essential before the patch can be committed?

[1] Fixed-length group absorption design note
https://www.postgresql.org/message-id/CAAAe_zAKAGKpK9iHx3ZSuG59sP93r5dfootqv5tCfaMt%3Dw6wzA%40mail.gmail.com
[2] FIRST/LAST navigation design note
https://www.postgresql.org/message-id/CAAAe_zCUrKGBgZdaazdO_v9QWHsS_1DXuP%3DrLeNhO3iwsHwwbg%40mail.gmail.com
[3] jit: Change the default to off
https://github.com/postgres/postgres/commit/7f8c88c2b872

Best regards,
Henson

2026년 4월 2일 (목) 오후 12:51, Henson Choi <assam258@gmail.com>님이 작성:

> Hi Tatsuo,
>
> Thank you for the review and the attached patch for 0005.
> I appreciate you taking the time to look at each patch
> carefully.
>
> Attached are 8 incremental patches on top of v46 (replacing
> the previous 6-patch set). 0001-0003 are unchanged. 0004 and
> 0005 incorporate your feedback below. 0006 is a new planner
> fix. Previous 0006 becomes 0007. 0008 is new.
>
> >   0004: Fix in-place modification of defineClause TargetEntry
>>
>> Probably we want to modify the comment above since it implies an
>> in-place modification? What about something like this? (Modifies ->
>> Replace)
>>
>>         /*
>>          * Replace an expression tree in each DEFINE clause so that all
>> Var
>>          * nodes's varno refers to OUTER_VAR.
>>          */
>>
>
> Good point, thank you. Applied in the updated 0004.
>
>
>> >   0005: Fix mark handling for last_value() under RPR
>>
>> I think instead we can set a mark at frameheadpos when seek type is
>> SEEK_TAIL and RPR is enabled. See attached patch.
>>
>
> Thank you for the patch. Your approach is cleaner -- setting
> the mark at frameheadpos is more direct than suppressing
> advancement. I've adopted it in the updated 0005.
>
>
>> >   0006: Implement 1-slot PREV/NEXT navigation for RPR
>>
>> Excellent! I will take a look at it. (it will take for a while).
>>
>
> No rush at all. In the meantime, while testing the PREV/NEXT
> patch in various query patterns, I found a planner issue.
> I've also added JIT support. Here is a summary of the new
> patches:
>
>
>   0006: Prevent removal of RPR window functions in unused
>         subquery outputs
>
>         Wrapping an RPR window query in a subquery with an
>         outer aggregate causes a crash:
>
>           SELECT count(*) FROM (
>             SELECT count(*) OVER w FROM ...
>             WINDOW w AS (... DEFINE A AS i > PREV(i))
>           ) t;
>
>         remove_unused_subquery_outputs() replaces unused subquery
>         target entries with NULL constants. When an RPR window
>         function's result is not referenced by the outer query,
>         this replacement eliminates all active window functions
>         for the WindowClause, causing the planner to omit the
>         WindowAgg node. DEFINE clause expressions containing
>         RPRNavExpr (PREV/NEXT) then lose their execution context,
>         leading to a crash.
>
>         The fix skips the NULL replacement for window functions
>         whose WindowClause has a defineClause. Even when the
>         window function result is unused, RPR pattern matching
>         (frame reduction) must still execute -- the WindowAgg
>         node must be preserved.
>
>         An alternative would be to let the planner remove the
>         window function but teach it to still generate the
>         WindowAgg node when defineClause is present, even with
>         no active window functions. That would be a more
>         targeted optimization but requires deeper planner
>         changes. Do you think the current approach is
>         sufficient, or would you prefer a different strategy?
>
>
>   0007: Implement 1-slot PREV/NEXT navigation for RPR
>         (unchanged from previous 0006)
>
>
>> > 2. LLVM JIT fallback (in 0006): The mid-expression slot swap
>>
>> I am not an expert of JIT, but for me, it sounds reasonable.  We can
>> enhance it later on.
>>
>
>   0008: Add JIT compilation support for RPR PREV/NEXT navigation
>
>         EEOP_OUTER_VAR normally uses slot pointers cached in the
>         entry block, but the mid-expression slot swap in
>         NAV_SET/RESTORE invalidates them. To avoid penalizing
>         existing expressions, the reload path is only generated
>         when RPR navigation opcodes are present in the
>         expression (compile-time decision). Expressions without
>         PREV/NEXT produce identical machine code as before.
>         NAV_SET/RESTORE use the standard build_EvalXFunc pattern.
>
>         A 100K-row PREV/NEXT test case that runs under
>         jit_above_cost=0 is included in rpr.sql.
>
> I would appreciate your thoughts on these when you have time.
>
>
> Best regards,
> Henson
>
>

  297. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-04-10T01:18:47Z

    Hi Henson,

>   0006: Fix DEFINE expression handling in RPR window
>         planning (revised)
> 
>         Integration tests in 0007 revealed two crashes:
>         (1) subquery wrapping with outer aggregate causes
>         WindowAgg removal when RPR window function output
>         is unused, (2) RPR and non-RPR windows coexisting
>         causes SIGSEGV from RPRNavExpr propagating to the
>         wrong WindowAgg. This version extends the fix to
>         extract only Var nodes via pull_var_clause()
>         instead of adding the whole DEFINE expression to
>         the targetlist. The allpaths.c guard is extended
>         to also preserve columns referenced by DEFINE
>         clauses.

I took a look at this and have a question.

--- a/src/backend/optimizer/path/allpaths.c
+++ b/src/backend/optimizer/path/allpaths.c
@@ -4750,6 +4750,74 @@ remove_unused_subquery_outputs(Query *subquery, RelOptInfo *rel,
 		if (contain_volatile_functions(texpr))
 			continue;
 
+		/*
+		 * If any RPR window clause references this column in its DEFINE
+		 * clause, don't remove it.  The DEFINE expression needs these columns
+		 * in the tuplestore slot for pattern matching evaluation, even if the
+		 * outer query doesn't reference them.
+		 */

Before this, there's a check in the function:

		if (tle->ressortgroupref || tle->resjunk)
			continue;

Below is the query in the regression test that is suppoed to trigger
the error.  In this case column "i" in the target list should have
resjunk flag to be set to true. So I thought the if statenment above
becomes true and the column i is not removed from subquery's target
list. Am I missing something?

The test case in the patch:

-- Subquery wrapping: RPR window inside outer aggregate.
-- Tests that WindowAgg is not removed by remove_unused_subquery_outputs()
-- when DEFINE clause contains PREV/NEXT.
--
-- PREV in DEFINE + outer aggregate

--EXPLAIN
SELECT count(*) FROM (
  SELECT count(*) OVER w FROM generate_series(1,10) i
  WINDOW w AS (
    ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
    PATTERN (A+)
    DEFINE A AS i > PREV(i)
  )
) t;

Also I think removal of the WindowAgg node is fine here because it's
not necessary to calculate the result of the sub query at
all. Actually the query above runs fine on v46. I guess some of the
incremental patches caused this to stop working. Can you elaborate?

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  298. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-04-10T08:51:54Z

    Hi Tatsuo,

Thank you for the careful review.

Before this, there's a check in the function:
>
>                 if (tle->ressortgroupref || tle->resjunk)
>                         continue;
>
> Below is the query in the regression test that is suppoed to trigger
> the error.  In this case column "i" in the target list should have
> resjunk flag to be set to true. So I thought the if statenment above
> becomes true and the column i is not removed from subquery's target
> list. Am I missing something?


That is a very good observation. For the test query you cited, the
resjunk check does indeed protect column "i", as you noted.

The allpaths.c guard is needed for a different scenario: when a Var
already exists in the inner SELECT as a non-resjunk entry, but the
outer query does not reference it. For example:

  SELECT count(*) FROM (
    SELECT i, count(*) OVER w FROM generate_series(1,10) i
    --     ^ in SELECT -> resjunk=false
    WINDOW w AS (
      ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
      PATTERN (A+)
      DEFINE A AS i > PREV(i)
    )
  ) t;
  -- outer uses only count(*) -> i is not in attrs_used

In this case, the Var addition logic in parse_rpr.c (pull_var_clause
+ dup check) finds that "i" already exists in the targetlist, so it
does not add a new resjunk entry. Since "i" is resjunk=false and the
outer query does not reference it, remove_unused_subquery_outputs()
would replace it with NULL, breaking DEFINE evaluation.

Changing the existing entry to resjunk=true is not an option either,
since it would alter the subquery's output schema -- the outer query
may reference that column, and at parse time we do not yet know
whether it will. The existing protection mechanisms (resjunk,
ressortgroupref, attrs_used, volatile check) cannot express
"referenced by DEFINE", since DEFINE is a new column reference path
that did not exist in PostgreSQL before. The allpaths.c guard
follows the same pattern as the existing SRF and volatile guards in
that function.

Also I think removal of the WindowAgg node is fine here because it's
> not necessary to calculate the result of the sub query at
> all. Actually the query above runs fine on v46. I guess some of the
> incremental patches caused this to stop working. Can you elaborate?
>

I agree -- for this specific query, removing the WindowAgg does not
change the count(*) result, since window functions do not change the
number of rows.

Regarding v46: the two bugs were actually latent from the initial
implementation, not caused by incremental patches. The old code
added the entire DEFINE expression (e.g., "i > PREV(i)") to the
query targetlist via findTargetlistEntrySQL99(). In most cases,
DEFINE conditions involve comparison operators, so the resulting
OpExpr did not match existing Var entries in the targetlist, and a
new resjunk entry was created. This accidentally prevented removal.
(In fact, a bare boolean DEFINE such as "DEFINE A AS flag" would
have exhibited the same removal bug even in v46, since
findTargetlistEntrySQL99 would reuse the existing entry instead
of creating a new resjunk one -- confirming that the issue was
latent in the original implementation, not introduced by recent
patches.)

However, that same approach caused the SIGSEGV (Bug 1): when RPR
and non-RPR windows coexist, the non-RPR WindowAgg inherits
targetlist entries containing RPRNavExpr nodes that it cannot
evaluate.

The two bugs are two sides of the same coin -- the old approach
prevented removal but caused SIGSEGV. Switching to Var-only
addition fixed the SIGSEGV but required the allpaths.c guard to
prevent removal.

As for more precise optimization (allowing removal when all
WindowFuncs for an RPR clause are unused), it would require
structural changes to the loop in remove_unused_subquery_outputs(),
which currently evaluates entries individually. Since this function
runs for all subqueries, not just RPR, such a change would need
broader testing.

The patch addresses two distinct issues: the parse_rpr.c change
(Var-only addition) prevents RPRNavExpr from propagating to
non-RPR WindowAgg nodes, while the allpaths.c guard prevents
incorrect column removal. Since both are needed, the guard
cannot simply be reverted. Would it make sense to keep it as-is
for now, and explore the more precise optimization as a separate
follow-up patch?

Best regards,
Henson

  299. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-04-10T09:35:15Z

    Hi Henson,

> Hi Tatsuo,
> 
> Thank you for the careful review.
> 
> Before this, there's a check in the function:
>>
>>                 if (tle->ressortgroupref || tle->resjunk)
>>                         continue;
>>
>> Below is the query in the regression test that is suppoed to trigger
>> the error.  In this case column "i" in the target list should have
>> resjunk flag to be set to true. So I thought the if statenment above
>> becomes true and the column i is not removed from subquery's target
>> list. Am I missing something?
> 
> 
> That is a very good observation. For the test query you cited, the
> resjunk check does indeed protect column "i", as you noted.
> 
> The allpaths.c guard is needed for a different scenario: when a Var
> already exists in the inner SELECT as a non-resjunk entry, but the
> outer query does not reference it. For example:
> 
>   SELECT count(*) FROM (
>     SELECT i, count(*) OVER w FROM generate_series(1,10) i
>     --     ^ in SELECT -> resjunk=false
>     WINDOW w AS (
>       ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>       PATTERN (A+)
>       DEFINE A AS i > PREV(i)
>     )
>   ) t;
>   -- outer uses only count(*) -> i is not in attrs_used
> 
> In this case, the Var addition logic in parse_rpr.c (pull_var_clause
> + dup check) finds that "i" already exists in the targetlist, so it
> does not add a new resjunk entry. Since "i" is resjunk=false and the
> outer query does not reference it, remove_unused_subquery_outputs()
> would replace it with NULL, breaking DEFINE evaluation.
> 
> Changing the existing entry to resjunk=true is not an option either,
> since it would alter the subquery's output schema -- the outer query
> may reference that column, and at parse time we do not yet know
> whether it will. The existing protection mechanisms (resjunk,
> ressortgroupref, attrs_used, volatile check) cannot express
> "referenced by DEFINE", since DEFINE is a new column reference path
> that did not exist in PostgreSQL before. The allpaths.c guard
> follows the same pattern as the existing SRF and volatile guards in
> that function.
> 
> Also I think removal of the WindowAgg node is fine here because it's
>> not necessary to calculate the result of the sub query at
>> all. Actually the query above runs fine on v46. I guess some of the
>> incremental patches caused this to stop working. Can you elaborate?
>>
> 
> I agree -- for this specific query, removing the WindowAgg does not
> change the count(*) result, since window functions do not change the
> number of rows.
> 
> Regarding v46: the two bugs were actually latent from the initial
> implementation, not caused by incremental patches. The old code
> added the entire DEFINE expression (e.g., "i > PREV(i)") to the
> query targetlist via findTargetlistEntrySQL99(). In most cases,
> DEFINE conditions involve comparison operators, so the resulting
> OpExpr did not match existing Var entries in the targetlist, and a
> new resjunk entry was created. This accidentally prevented removal.
> (In fact, a bare boolean DEFINE such as "DEFINE A AS flag" would
> have exhibited the same removal bug even in v46, since
> findTargetlistEntrySQL99 would reuse the existing entry instead
> of creating a new resjunk one -- confirming that the issue was
> latent in the original implementation, not introduced by recent
> patches.)
> 
> However, that same approach caused the SIGSEGV (Bug 1): when RPR
> and non-RPR windows coexist, the non-RPR WindowAgg inherits
> targetlist entries containing RPRNavExpr nodes that it cannot
> evaluate.
> 
> The two bugs are two sides of the same coin -- the old approach
> prevented removal but caused SIGSEGV. Switching to Var-only
> addition fixed the SIGSEGV but required the allpaths.c guard to
> prevent removal.
> 
> As for more precise optimization (allowing removal when all
> WindowFuncs for an RPR clause are unused), it would require
> structural changes to the loop in remove_unused_subquery_outputs(),
> which currently evaluates entries individually. Since this function
> runs for all subqueries, not just RPR, such a change would need
> broader testing.
> 
> The patch addresses two distinct issues: the parse_rpr.c change
> (Var-only addition) prevents RPRNavExpr from propagating to
> non-RPR WindowAgg nodes, while the allpaths.c guard prevents
> incorrect column removal. Since both are needed, the guard
> cannot simply be reverted.

Thank you for the detailed explanation. That makes sense.

> Would it make sense to keep it as-is
> for now, and explore the more precise optimization as a separate
> follow-up patch?

I agree keep it as is. I think priority of "more precise optimization"
is low for now. We could explore when we have time. What do you think?

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  300. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-04-12T07:27:26Z

    Hi Tatsuo,

When I started this work a little over three months ago, I had
no idea it would grow into what it is today. There is still
much to refine, but I would not have come this far without
your guidance and support along the way.

This series completes RPR navigation support, including
PREV/NEXT, FIRST/LAST, and compound navigation such as
PREV(FIRST(col)).

Attached are 31 incremental patches on top of v46 (replacing
the previous 15-patch set). Here is a summary of the changes
from the previous version:


  0001-0007: Unchanged from the previous set.

  0008: Replace reduced frame map with single match result
        (revised: int -> int64 type widening for
        row_is_in_reduced_frame() and related variables,
        cascading from 0025)

  0009: Add fixed-length group absorption for RPR
        (unchanged, design note [1])

  0010: Rename rpr_explain test views to descriptive
        names (unchanged)


  0011: Fix quote_identifier() for RPR pattern variable
        name deparse (new)

        Pattern variable names that match SQL keywords were
        not being quoted in pg_get_viewdef() output, causing
        the deparsed view to fail on re-parse.

  0012: Fix execRPR.o ordering in executor Makefile to
        match meson.build (new)

        Ensures consistent build file ordering between Make
        and Meson build systems.

  0013: Remove unused force_colno parameter from RPR
        deparse functions (new)

        Dead parameter cleanup in get_rpr_pattern() and
        get_rpr_define() in ruleutils.c.

  0014: Add CHECK_FOR_INTERRUPTS to RPR context cleanup
        and finalize loops (new)

        Extends interrupt checking to ExecRPRCleanupContexts()
        and ExecRPRFinalizeMatch() loops, complementing the
        earlier additions in 0002 and 0003.

  0015: Narrow variable scope in ExecInitWindowAgg DEFINE
        clause loop (new)

        Moves local variable declarations into their
        respective loop bodies for clarity.

  0016: Normalize RPR element flag macros to return bool
        (new)

        RPR_ELEM_IS_* macros now return bool instead of the
        raw bitmask value, consistent with PostgreSQL style.


  0017: Implement 1-slot PREV/NEXT navigation for RPR
        (was 0011, context changes from 0008 int64)

  0018: Add JIT compilation support for RPR PREV/NEXT
        (was 0012, unchanged)

  0019: Add tuplestore trim optimization for RPR PREV
        navigation (was 0013, context changes from 0008
        int64)

  0020: Update RPR code comments to reflect 1-slot
        navigation model (was 0014, context only)


  0021: Enable JIT compilation for PREV/NEXT navigation
        tests in RPR (new)

        Adds test coverage for JIT-compiled PREV/NEXT
        expressions under jit_above_cost=0, verifying
        that the slot reload path produces correct results.

  0022: Add 2-arg PREV/NEXT test for row pattern
        navigation with host variable (new)

        Tests the PREV(value, offset) form where offset
        is a host variable (prepared statement parameter),
        exercising the RPR_NAV_OFFSET_NEEDS_EVAL path.

  0023: Add Nav Mark Lookback to EXPLAIN and fix
        compute_nav_max_offset() (new)

        EXPLAIN now displays "Nav Mark Lookback: N" for
        RPR windows with PREV navigation, showing how
        many rows the tuplestore retains. Also fixes
        compute_nav_max_offset() to correctly handle
        non-constant offset expressions.


  0024: Implement FIRST/LAST and compound navigation
        for RPR (was 0015, completed)

        The previous WIP patch is now complete with:

        - Compound navigation: PREV(FIRST(col)),
          NEXT(LAST(col, N)), etc. per SQL standard 5.6.4.
          Flattened into a single RPRNavExpr node with
          two offset values (inner position + outer offset).

        - RPRNavOffsetKind enum replaces sentinel macros
          for offset classification (CONSTANT, NEEDS_EVAL,
          FIRST_BASED).

        - Per-context DEFINE re-evaluation for match_start-
          dependent expressions (FIRST, LAST with offset),
          tracked via defineMatchStartDependent bitmapset
          computed by the planner.

        - Tuplestore mark handling extended for FIRST-based
          navigation: mark position accounts for both
          backward reach (PREV) and forward-from-match-start
          reach (FIRST).

        - Documentation added to func-window.sgml.

        See the design note [2] for the absorption safety
        analysis and evaluation sharing rules.


  0025: Guard against int64 overflow in RPR bounded frame
        end computation (new)

        Widens frame position variables from int to int64
        in row_is_in_reduced_frame() and related functions.
        Prevents overflow when frame end position exceeds
        INT_MAX in large partitions.

  0026: Fix RPR error message style: hint format,
        terminology, capitalization (new)

        Aligns RPR error messages with PostgreSQL conventions:
        lowercase first word, consistent use of "row pattern"
        terminology, HINT format corrections.

  0027: Fix comment typos, grammar, and inaccuracies in
        RPR code (new)

        Comment-only changes across execRPR.c, parse_rpr.c,
        nodeWindowAgg.c, and ruleutils.c. No functional
        changes.

  0028: Fix RPR documentation: synopsis, grammar, and
        terminology (new)

        Updates select.sgml, advanced.sgml, and
        func-window.sgml for consistent terminology
        and corrected synopsis examples.

  0029: Fix nav_slot pass-by-ref dangling pointer in RPR
        navigation (new)

        When a DEFINE expression contains multiple navigation
        calls targeting different positions (e.g.,
        PREV(x,1) > PREV(x,2)), the second call re-fetches
        nav_slot, freeing the previous tuple via pfree. Any
        pass-by-ref datum extracted from the first navigation
        becomes a dangling pointer. Fix by copying pass-by-ref
        results into per-tuple memory in the RESTORE step.

  0030: Add inline comments for complex RPR algorithms and
        design notes (new)

        Adds explanatory comments to key algorithms in
        execRPR.c and rpr.c (planner). Covers NFA absorption
        logic, match evaluation flow, navigation mark
        computation, and DEFINE re-evaluation decisions.
        Comment-only changes, no functional changes.

  0031: Remove unused include and fix header ordering in RPR
        files (new)

        Removes an unnecessary include from execExprInterp.c,
        drops a redundant nodeWindowAgg.h include from
        nodeWindowAgg.c, and fixes header ordering in
        parse_rpr.c to match PostgreSQL convention.


Changes from the previous version:

  - 0015 (FIRST/LAST) is now complete as 0024, with compound
    navigation, RPRNavOffsetKind, defineMatchStartDependent,
    and full test coverage.

  - 16 new patches (0011-0016, 0021-0023, 0025-0031) added
    since the previous set. These are all quality improvements:
    bug fixes, safety guards, style normalization, comments,
    and test coverage.

  - New features: compound navigation in 0024 (was "not
    yet done" in previous set), Nav Mark Lookback in
    EXPLAIN output (0023). All other new patches are
    quality improvements.

I am planning to submit PREFIX pattern absorption [3]
as a separate patch after this set is committed. The
changes are confined to the Absorb phase with no
external dependencies, so it can be submitted
independently. The design itself has some complexity
(shadow path tracking), and I would like to refine it
further until I am fully confident in the approach.

With this set I have covered the features I could
foresee. Whether anything else belongs in scope is
your call. From here, I think the emphasis should
shift from adding features to raising overall
quality.
If there are directions you think should be explored,
I am happy to work on them. Otherwise, I plan to focus
on code review, test coverage, and stabilization, and
will continue to submit bug fixes and test additions
as I find them.

I also very much welcome refactoring or restructuring
suggestions from a committer's maintainability
perspective. If anything in the current code will be
painful to maintain long-term, I would much rather
address it now than leave it for after commit.

For stabilization, I am thinking of the same approach
as in early January: gcov analysis, Valgrind runs, and
expanding the test suite.

**WITH THE PLANNER**, the biggest risk is what we do
not know we are missing -- there are too many possible
feature interactions to enumerate up front. To surface
these, I plan to run RPR-only tests under gcov and
review the uncovered planner paths: any line that RPR
could plausibly reach but does not becomes a candidate
risk, and a candidate for a new test. That turns the
unknown into a concrete review list rather than a guess
about which interactions to try. Does that seem like a
good direction?

If anything comes to mind that should be added or
fixed -- at any time -- please let me know.

[1] Fixed-length group absorption design note
https://www.postgresql.org/message-id/CAAAe_zAKAGKpK9iHx3ZSuG59sP93r5dfootqv5tCfaMt%3Dw6wzA%40mail.gmail.com
[2] FIRST/LAST navigation design note
https://www.postgresql.org/message-id/CAAAe_zCUrKGBgZdaazdO_v9QWHsS_1DXuP%3DrLeNhO3iwsHwwbg%40mail.gmail.com
[3] PREFIX pattern absorption design note
https://www.postgresql.org/message-id/CAAAe_zDq7R8CaDfmh8C%2BH3_639Y5LtJD%2B2Z%3D1txDt%3DvaOr90rQ%40mail.gmail.com

Best regards,
Henson

  301. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-04-13T22:52:25Z

    Hi Henson,

> Hi Tatsuo,
> 
> When I started this work a little over three months ago, I had
> no idea it would grow into what it is today. There is still
> much to refine, but I would not have come this far without
> your guidance and support along the way.

I had no idea too :-) I was struggled with RPR develpment issues,
especially on an NFA engine and navigation operations (PREV/NEXT
etc.). You have solved both of the issues.

> This series completes RPR navigation support, including
> PREV/NEXT, FIRST/LAST, and compound navigation such as
> PREV(FIRST(col)).
> 
> Attached are 31 incremental patches on top of v46 (replacing
> the previous 15-patch set). Here is a summary of the changes
> from the previous version:
> 
> 
>   0001-0007: Unchanged from the previous set.

Here are the review for 0001-0007. I will continue to review rest of
the patches.

0001 was already reviewed (LGTM).
0002-0003 looks good to me.
0004 was already reviewed (LGTM).
0005 thanks for review.
0006

+		 * If it's a window function referencing a window clause with RPR (Row
+		 * Pattern Recognition), don't remove it.  Even when the window

As "RPR" already appears in the prevous comment block above, the "RPR"
appearing in the fist comment block should be "RPR (Row Pattern
Recognition)" and "RPR" here should be without "(Row Pattern
Recognition)".

0007:

diff --git a/src/test/regress/expected/rpr_integration.out b/src/test/regress/expected/rpr_integration.out
new file mode 100644
index 00000000000..a21ac5a8588
--- /dev/null
+++ b/src/test/regress/expected/rpr_integration.out
+-- Non-RPR: frame is optimized (RANGE -> ROWS conversion, etc.)
+-- RPR: frame must stay as specified (ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)

For these 2 lines I feel a little bit hard to parse. Probably better
to be written as complete sentences, rather than bullet points. For
example,

+-- Non-RPR case: frame can be optimized (RANGE -> ROWS conversion, etc.)
+-- RPR case: frame must stay as specified (ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)

Better to have a blanck line here for readability.

+-- Non-RPR window with default frame -> frame optimization applied

+-- ============================================================
+-- B1. RPR + CTE
+-- ============================================================
+-- CTE with RPR, outer query aggregates

Better to comment what we want to test (or expect).

+-- Multiple CTE references

Better to comment what we want to test (or expect).

+-- ============================================================
+-- B2. RPR + JOIN
+-- ============================================================

Better to comment what we want to test (or expect).

+-- ============================================================
+-- B3. RPR + Set operations
+-- ============================================================

Better to comment what we want to test (or expect).

+-- ============================================================
+-- B8. RPR + Incremental sort
+-- ============================================================
+-- Incremental sort may be used when data is partially sorted.
+CREATE INDEX rpr_integ_id_idx ON rpr_integ (id);
+SET enable_seqscan = off;
+EXPLAIN (COSTS OFF)
+SELECT id, val, count(*) OVER w AS cnt
+FROM rpr_integ
+WINDOW w AS (ORDER BY id
+    ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+    PATTERN (A B+)
+    DEFINE B AS val > PREV(val));
+                                  QUERY PLAN                                   
+-------------------------------------------------------------------------------
+ WindowAgg
+   Window: w AS (ORDER BY id ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
+   Pattern: a b+
+   ->  Index Scan using rpr_integ_id_idx on rpr_integ
+(4 rows)

No incremental sort in this plan?

+-- ============================================================
+-- B9. RPR + Volatile function in DEFINE
+-- ============================================================
+-- Volatile functions must be evaluated per-row, not optimized away.

Can you elaborate how the test below confirm random() is not optimized
away?

+SELECT id, val, count(*) OVER w AS cnt
+FROM rpr_integ
+WINDOW w AS (ORDER BY id
+    ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
+    PATTERN (A B+)
+    DEFINE B AS val > PREV(val) AND random() >= 0.0)
+ORDER BY id;
+ id | val | cnt 
+----+-----+-----
+  1 |  10 |   2
+  2 |  20 |   0
+  3 |  15 |   2
+  4 |  25 |   0
+  5 |   5 |   3
+  6 |  30 |   0
+  7 |  35 |   0
+  8 |  20 |   3
+  9 |  40 |   0
+ 10 |  45 |   0
+(10 rows)

0008 and beyond will be reviewd in separate emails.

> I am planning to submit PREFIX pattern absorption [3]
> as a separate patch after this set is committed. The
> changes are confined to the Absorb phase with no
> external dependencies, so it can be submitted
> independently. The design itself has some complexity
> (shadow path tracking), and I would like to refine it
> further until I am fully confident in the approach.

Ok.

> With this set I have covered the features I could
> foresee. Whether anything else belongs in scope is
> your call. From here, I think the emphasis should
> shift from adding features to raising overall
> quality.

Agreed.

> If there are directions you think should be explored, I am happy to
> work on them. Otherwise, I plan to focus on code review, test
> coverage, and stabilization, and will continue to submit bug fixes
> and test additions as I find them.

One thing I want to discuss is, how to deal with the cases when
volatile functions are used in the DEFINE clause. As far as I know,
the SQL standard does not prohibit to use volatile functions there,
but we may need to prohibit to use it for our implementation reasons.

For reference, I studied how non RPR part in WindowAggs.c handles
volatile functions.

1. When searching for identical window functions, if they are volatile
functions, they are considered different window functions.

2. When evaluating the frame offset expression, it is evaluated only once
each time a scan/rescan occurs.

3. When checking whether moving aggregation is available, the
volatility of the window function is checked. If it is volatile,
moving aggregation is disabled.

> I also very much welcome refactoring or restructuring
> suggestions from a committer's maintainability
> perspective. If anything in the current code will be
> painful to maintain long-term, I would much rather
> address it now than leave it for after commit.

Ok, I will let you know when I find anything.

> For stabilization, I am thinking of the same approach
> as in early January: gcov analysis, Valgrind runs, and
> expanding the test suite.
> 
> **WITH THE PLANNER**, the biggest risk is what we do
> not know we are missing -- there are too many possible
> feature interactions to enumerate up front. To surface
> these, I plan to run RPR-only tests under gcov and
> review the uncovered planner paths: any line that RPR
> could plausibly reach but does not becomes a candidate
> risk, and a candidate for a new test. That turns the
> unknown into a concrete review list rather than a guess
> about which interactions to try. Does that seem like a
> good direction?

Sounds like a good direction.

> If anything comes to mind that should be added or
> fixed -- at any time -- please let me know.

Sure, I will.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  302. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-04-17T08:10:22Z

    Here is a review for 0008.

The patch does not apply using git apply.
$ git apply /usr/local/src/pgsql/RPR/make_rpr_patches/v47/input_patches/nocfbot-0008-Replace-reduced-frame-map-with-single-match-result.txt 
error: patch failed: src/backend/executor/nodeWindowAgg.c:247
error: src/backend/executor/nodeWindowAgg.c: patch does not apply
error: patch failed: src/include/nodes/execnodes.h:2694
error: src/include/nodes/execnodes.h: patch does not apply

I applied it using patch command.

$ patch -p1 < /usr/local/src/pgsql/RPR/make_rpr_patches/v47/input_patches/nocfbot-0008-Replace-reduced-frame-map-with-single-match-result.txt 
patching file src/backend/executor/execRPR.c
patching file src/backend/executor/nodeWindowAgg.c
Hunk #1 succeeded at 235 with fuzz 2 (offset -12 lines).
Hunk #2 succeeded at 1017 (offset -15 lines).
Hunk #3 succeeded at 1041 (offset -15 lines).
Hunk #4 succeeded at 1060 (offset -15 lines).
Hunk #5 succeeded at 1340 (offset 4 lines).
Hunk #6 succeeded at 1571 (offset 11 lines).
Hunk #7 succeeded at 2355 (offset 11 lines).
Hunk #8 succeeded at 2464 (offset 11 lines).
Hunk #9 succeeded at 2991 (offset 33 lines).
Hunk #10 succeeded at 3600 (offset -37 lines).
Hunk #11 succeeded at 3900 (offset -37 lines).
Hunk #12 succeeded at 3914 (offset -37 lines).
Hunk #13 succeeded at 3925 (offset -37 lines).
Hunk #14 succeeded at 3938 (offset -37 lines).
Hunk #15 succeeded at 3948 (offset -37 lines).
Hunk #16 succeeded at 4044 (offset -37 lines).
Hunk #17 succeeded at 4065 (offset -37 lines).
Hunk #18 succeeded at 4141 (offset -37 lines).
Hunk #19 succeeded at 4657 (offset -35 lines).
patching file src/include/nodes/execnodes.h
Hunk #2 succeeded at 2698 with fuzz 2 (offset 2 lines).
patching file src/test/regress/expected/rpr_explain.out

From: Henson Choi <assam258@gmail.com>
Subject: Re: Row pattern recognition
Date: Sun, 12 Apr 2026 16:27:26 +0900
Message-ID: <CAAAe_zB7rAEJtT6hXgF85=_Tj8Nti45ZHbQw26gxTF2DBs3hJw@mail.gmail.com>

> From 59e99c6b9322f402df560bc693863491487e12db Mon Sep 17 00:00:00 2001
> From: Henson Choi <assam258@gmail.com>
> Date: Thu, 2 Apr 2026 14:09:12 +0900
> Subject: [PATCH] Replace reduced frame map with single match result
> 
> The reduced frame map was a per-row byte array tracking match status.
> Since rows are processed sequentially and only one match is active
> at a time, replace it with four scalar fields: valid, matched,
> start, and length.

You are right, we don't need to have the map as an array.

> Also distinguish empty matches (FIN reached with zero rows consumed)
> from unmatched rows via RF_EMPTY_MATCH, counted as matched in NFA
> statistics.

Good enhancement.

> Widen row_is_in_reduced_frame() return type from int to int64,
> since it returns rpr_match_length which is int64.

Good point.

> ---
>  src/backend/executor/execRPR.c            |  56 +++---
>  src/backend/executor/nodeWindowAgg.c      | 233 +++++++++-------------
>  src/include/nodes/execnodes.h             |  21 +-
>  src/test/regress/expected/rpr_explain.out |   8 +-
>  4 files changed, 132 insertions(+), 186 deletions(-)
> 
> diff --git a/src/backend/executor/execRPR.c b/src/backend/executor/execRPR.c
> index 58f9da0b814..1cbe8e14780 100644
> --- a/src/backend/executor/execRPR.c
> +++ b/src/backend/executor/execRPR.c
> @@ -549,7 +549,7 @@
>   *
>   *   (1) Find or create a context for the target row
>   *   (2) Enter the row processing loop
> - *   (3) After the loop ends, record the result in reduced_frame_map
> + *   (3) After the loop ends, record the match result
>   *
>   * Pseudocode of the row processing loop:
>   *
> @@ -923,18 +923,19 @@
>   * Chapter X  Match Result Processing
>   * ============================================================================
>   *
> - * X-1. Reduced Frame Map
> + * X-1. Match Result
>   *
> - * RPR match results are recorded in a byte array called reduced_frame_map.
> - * One byte is allocated per row, and the value is one of the following:
> + * RPR tracks the current match result as a single entry in WindowAggState
> + * with four fields: rpr_match_valid, rpr_match_matched, rpr_match_start,
> + * and rpr_match_length.  When valid is true, the entry describes the

"When valid is true" sounds a little bit confusing for readers. I
think "When rpr_match_valid is true" is cleaner, although it adds more
characters. Same thing can be said to "matchded", "valid" and "length".

> + * match result for the position at rpr_match_start: matched indicates
> + * success or failure, and length gives the number of rows consumed.
> + * A match with length 0 represents an empty match (pattern matched but
> + * consumed no rows).  When valid is false, the position has not been
> + * evaluated yet (RF_NOT_DETERMINED).
>   *
> - *   RF_NOT_DETERMINED (0)  Not yet processed
> - *   RF_FRAME_HEAD     (1)  Start row of the match
> - *   RF_SKIPPED        (2)  Interior row of the match (skipped in frame)
> - *   RF_UNMATCHED      (3)  Match failure
> - *
> - * The window function references this map to determine frame inclusion for
> - * each row.
> + * The window function calls get_reduced_frame_status() to look up a
> + * row's status against the current match result.

This may be misleading since get_reduced_frame_status() is not an
exposed public API to a window function. What about something like
"Row's status against the current match result can be obtained by
calling get_reduced_frame_status()"?

In execRPR.c, documentation occupies 1445 lines out of 3052 lines. I
don't see any other .[ch] files, half of it is taken up by
documentation. I suggest to create something like "README.rpr" and
move the documentation part to it.

> diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
> index aed7cbef99a..dca2de570e8 100644
> --- a/src/backend/executor/nodeWindowAgg.c
> +++ b/src/backend/executor/nodeWindowAgg.c

No comment to nodeWindowAgg.c patch. This part looks good to me.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  303. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-04-17T10:13:03Z

    Hi Henson,

Note that applying 0001-0006 produced compiler warning.

execExprInterp.c: In function ‘ExecEvalRPRNavSet’:
execExprInterp.c:6005:23: warning: ‘target_pos’ may be used uninitialized [-Wmaybe-uninitialized]
 6005 |         target_slot = ExecRPRNavGetSlot(winstate, target_pos);
      |                       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
execExprInterp.c:5959:25: note: ‘target_pos’ was declared here
 5959 |         int64           target_pos;
      |                         ^~~~~~~~~~

But regression test was ok.
However, after applying 0007, regression test failed.

not ok 196   + rpr_integration                           514 ms
# (test process exited with exit code 2)
# parallel group (9 tests):  jsonpath jsonpath_encoding jsonb_jsonpath sqljson_queryfuncs json_encoding json sqljson_jsontable jsonb sqljson
not ok 197   + json                                       29 ms
# (test process exited with exit code 2)
not ok 198   + jsonb                                      29 ms
# (test process exited with exit code 2)
[snip]

From rpr_integration.out:

-- Verify both produce different results
SELECT
    id, val,
    count(*) OVER w_rpr AS rpr_cnt,
    count(*) OVER w_normal AS normal_cnt
FROM rpr_integ
WINDOW
    w_rpr AS (ORDER BY id
        ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
        PATTERN (A B+)
        DEFINE B AS val > PREV(val)),
    w_normal AS (ORDER BY id
        ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
ORDER BY id;
server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
connection to server was lost

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  304. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-04-17T12:08:12Z

    Hi Tatsuo,

Thank you for the feedback.

Note that applying 0001-0006 produced compiler warning.
>
> execExprInterp.c: In function ‘ExecEvalRPRNavSet’:
> execExprInterp.c:6005:23: warning: ‘target_pos’ may be used uninitialized
> [-Wmaybe-uninitialized]
>  6005 |         target_slot = ExecRPRNavGetSlot(winstate, target_pos);
>       |                       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> execExprInterp.c:5959:25: note: ‘target_pos’ was declared here
>  5959 |         int64           target_pos;
>       |                         ^~~~~~~~~~
>

`ExecRPRNavGetSlot()`, along with the enclosing `ExecEvalRPRNavSet()`
and its `int64 target_pos` declaration, is introduced by nocfbot-0017
(Implement 1-slot PREV/NEXT navigation for RPR), not at the 0001-0006
level.  Could you double-check the actual applied range on your side?
The function simply does not exist at 0001-0006.

Since the function is not present at 0001-0006, I suspect the
applied patch set on your side is in a different state than
expected.  Could you verify the range and contents of the patches
actually applied?

But regression test was ok.
> However, after applying 0007, regression test failed.
>
> not ok 196   + rpr_integration                           514 ms
> # (test process exited with exit code 2)
>

nocfbot-0006 was written specifically to prevent the crashes that
0007's tests expose, so with 0001-0007 applied in order the issue
should already be resolved.  On my authoritative base (cfbot/4460) all
rpr_integration tests pass through 0008.  I had been routinely
building only on macOS ARM64 recently, so I just re-verified on
Rocky 9 and Ubuntu 24 AMD64 as well; no build warning and no
rpr_integration failure on either.

For reference, here is my working tree layout.  Titles only for my
post-base commits (hashes on my side won't match yours); base
hashes are public and directly comparable on your side:

  [post-base, newest first]
   31. Remove unused include and fix header ordering in RPR files
(origin/RPR)
   30. Add inline comments for complex RPR algorithms and design notes
   29. Fix nav_slot pass-by-ref dangling pointer in RPR navigation
   28. Fix RPR documentation: synopsis, grammar, and terminology
   27. Fix comment typos, grammar, and inaccuracies in RPR code
   26. Fix RPR error message style: hint format, terminology, capitalization
   25. Guard against int64 overflow in RPR bounded frame end computation
   24. Implement FIRST/LAST and compound navigation for RPR
   23. Add Nav Mark Lookback to EXPLAIN and fix compute_nav_max_offset()
   22. Add 2-arg PREV/NEXT test for row pattern navigation with host
variable
   21. Enable JIT compilation for PREV/NEXT navigation tests in RPR
   20. Update RPR code comments to reflect 1-slot navigation model
   19. Add tuplestore trim optimization for RPR PREV navigation
   18. Add JIT compilation support for RPR PREV/NEXT navigation
   17. Implement 1-slot PREV/NEXT navigation for RPR
   16. Normalize RPR element flag macros to return bool
   15. Narrow variable scope in ExecInitWindowAgg DEFINE clause loop
   14. Add CHECK_FOR_INTERRUPTS to RPR context cleanup and finalize loops
   13. Remove unused force_colno parameter from RPR deparse functions
   12. Fix execRPR.o ordering in executor Makefile to match meson.build
   11. Fix quote_identifier() for RPR pattern variable name deparse
   10. Rename rpr_explain test views from sequential numbers to descriptive
names
    9. Add fixed-length group absorption for RPR
    8. Replace reduced frame map with single match result
    7. Add RPR planner integration tests
    6. Fix DEFINE expression handling in RPR window planning
    5. Fix mark handling for last_value() under RPR
    4. Fix in-place modification of defineClause TargetEntry in setrefs.c
    3. Add CHECK_FOR_INTERRUPTS() to nfa_try_absorb_context() loop
    2. Add CHECK_FOR_INTERRUPTS() to nfa_add_state_unique() for state
explosion patterns
    1. Remove unused regex/regex.h include from nodeWindowAgg.c

  [base, public hashes]
  732acf9b7c6 (cfbot/cf/4460) [CF 4460] v46 - Implement row pattern
recognition feature
  6eb8841afaf Row pattern recognition patch (typedefs.list).
  5213dbb3c4f Row pattern recognition patch (tests: expected).
  bdc3a17982b Row pattern recognition patch (tests: SQL, data).
  596bcc434b5 Row pattern recognition patch (docs).
  e02872fd59c Row pattern recognition patch (executor and commands).
  742c6eb673e Row pattern recognition patch (planner).
  8514ca64f2d Row pattern recognition patch (rewriter).
  d26741d4f1d Row pattern recognition patch (parse/analysis).
  f85d2ee8f4a Row pattern recognition patch for raw parser.
  e484b0eea61 (origin/RPR-base, RPR-base) Fix two issues in fast-path FK
check introduced by commit 2da86c1ef9

Could you send me the equivalent `git log` for your working tree
covering the full v46 application and the commit immediately
preceding it?  In particular, could you also check whether the
recent master commit titled "Fix integer overflow in
nodeWindowAgg.c" (backpatched to 14) is included in your base?  It
touches the same ROWS/GROUPS frame boundary code path that RPR
interacts with, so its presence or absence will help narrow things
down.  Once I can match the same base locally, I will reproduce and
include the fix in the next revision.

A quick status update:

  - nocfbot-0006: self-review complete, not yet sent (comment tweak per
review)
  - nocfbot-0007: under self-review (comment details per review)
  - nocfbot-0008: in progress

I would like to resolve the crash first before sending the next
revision, so that the full set goes out together once our bases are
aligned and A3 passes on your side.

One process question: so far I have been folding review feedback
into the relevant existing patches (i.e. amending nocfbot-0008 to
reflect your comments on it).  Would you prefer that I instead
append the review fixes as new patches at the end of the series
(e.g. nocfbot-0032, 0033, ...), so the history of reviewed changes
stays visible?  Happy to follow whichever workflow you find easier
to review.

Regards,
Henson

  305. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-04-17T13:47:08Z

    Hi Henson,

> Hi Tatsuo,
> 
> Thank you for the feedback.
> 
> Note that applying 0001-0006 produced compiler warning.
>>
>> execExprInterp.c: In function ‘ExecEvalRPRNavSet’:
>> execExprInterp.c:6005:23: warning: ‘target_pos’ may be used uninitialized
>> [-Wmaybe-uninitialized]
>>  6005 |         target_slot = ExecRPRNavGetSlot(winstate, target_pos);
>>       |                       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> execExprInterp.c:5959:25: note: ‘target_pos’ was declared here
>>  5959 |         int64           target_pos;
>>       |                         ^~~~~~~~~~
>>
> 
> `ExecRPRNavGetSlot()`, along with the enclosing `ExecEvalRPRNavSet()`
> and its `int64 target_pos` declaration, is introduced by nocfbot-0017
> (Implement 1-slot PREV/NEXT navigation for RPR), not at the 0001-0006
> level.  Could you double-check the actual applied range on your side?
> The function simply does not exist at 0001-0006.
> 
> Since the function is not present at 0001-0006, I suspect the
> applied patch set on your side is in a different state than
> expected.  Could you verify the range and contents of the patches
> actually applied?

> But regression test was ok.
>> However, after applying 0007, regression test failed.
>>
>> not ok 196   + rpr_integration                           514 ms
>> # (test process exited with exit code 2)
>>
> 
> nocfbot-0006 was written specifically to prevent the crashes that
> 0007's tests expose, so with 0001-0007 applied in order the issue
> should already be resolved.  On my authoritative base (cfbot/4460) all
> rpr_integration tests pass through 0008.  I had been routinely
> building only on macOS ARM64 recently, so I just re-verified on
> Rocky 9 and Ubuntu 24 AMD64 as well; no build warning and no
> rpr_integration failure on either.

You are right. I checked and found the 0006 patch I applied was wrong
one.

nocfbot-0006-Implement-1-slot-PREV-NEXT-navigation-for-RPR.txt:

This should have been:
nocfbot-0006-Fix-DEFINE-expression-handling-RPR-window-planning.txt

Sorry for confusion.

> For reference, here is my working tree layout.  Titles only for my
> post-base commits (hashes on my side won't match yours); base
> hashes are public and directly comparable on your side:
> 
>   [post-base, newest first]
>    31. Remove unused include and fix header ordering in RPR files
> (origin/RPR)
>    30. Add inline comments for complex RPR algorithms and design notes
>    29. Fix nav_slot pass-by-ref dangling pointer in RPR navigation
>    28. Fix RPR documentation: synopsis, grammar, and terminology
>    27. Fix comment typos, grammar, and inaccuracies in RPR code
>    26. Fix RPR error message style: hint format, terminology, capitalization
>    25. Guard against int64 overflow in RPR bounded frame end computation
>    24. Implement FIRST/LAST and compound navigation for RPR
>    23. Add Nav Mark Lookback to EXPLAIN and fix compute_nav_max_offset()
>    22. Add 2-arg PREV/NEXT test for row pattern navigation with host
> variable
>    21. Enable JIT compilation for PREV/NEXT navigation tests in RPR
>    20. Update RPR code comments to reflect 1-slot navigation model
>    19. Add tuplestore trim optimization for RPR PREV navigation
>    18. Add JIT compilation support for RPR PREV/NEXT navigation
>    17. Implement 1-slot PREV/NEXT navigation for RPR
>    16. Normalize RPR element flag macros to return bool
>    15. Narrow variable scope in ExecInitWindowAgg DEFINE clause loop
>    14. Add CHECK_FOR_INTERRUPTS to RPR context cleanup and finalize loops
>    13. Remove unused force_colno parameter from RPR deparse functions
>    12. Fix execRPR.o ordering in executor Makefile to match meson.build
>    11. Fix quote_identifier() for RPR pattern variable name deparse
>    10. Rename rpr_explain test views from sequential numbers to descriptive
> names
>     9. Add fixed-length group absorption for RPR
>     8. Replace reduced frame map with single match result
>     7. Add RPR planner integration tests
>     6. Fix DEFINE expression handling in RPR window planning
>     5. Fix mark handling for last_value() under RPR
>     4. Fix in-place modification of defineClause TargetEntry in setrefs.c
>     3. Add CHECK_FOR_INTERRUPTS() to nfa_try_absorb_context() loop
>     2. Add CHECK_FOR_INTERRUPTS() to nfa_add_state_unique() for state
> explosion patterns
>     1. Remove unused regex/regex.h include from nodeWindowAgg.c
> 
>   [base, public hashes]
>   732acf9b7c6 (cfbot/cf/4460) [CF 4460] v46 - Implement row pattern
> recognition feature
>   6eb8841afaf Row pattern recognition patch (typedefs.list).
>   5213dbb3c4f Row pattern recognition patch (tests: expected).
>   bdc3a17982b Row pattern recognition patch (tests: SQL, data).
>   596bcc434b5 Row pattern recognition patch (docs).
>   e02872fd59c Row pattern recognition patch (executor and commands).
>   742c6eb673e Row pattern recognition patch (planner).
>   8514ca64f2d Row pattern recognition patch (rewriter).
>   d26741d4f1d Row pattern recognition patch (parse/analysis).
>   f85d2ee8f4a Row pattern recognition patch for raw parser.
>   e484b0eea61 (origin/RPR-base, RPR-base) Fix two issues in fast-path FK
> check introduced by commit 2da86c1ef9
> 
> Could you send me the equivalent `git log` for your working tree
> covering the full v46 application and the commit immediately
> preceding it?

Here it is:

t-ishii$ git log --oneline -10|cat
6dd0765463a Row pattern recognition patch (typedefs.list).
009b38f4d2b Row pattern recognition patch (tests: expected).
07864e40e36 Row pattern recognition patch (tests: SQL, data).
55b467849eb Row pattern recognition patch (docs).
28a1ff7e744 Row pattern recognition patch (executor and commands).
728b1fdf106 Row pattern recognition patch (planner).
c388e3e7eb0 Row pattern recognition patch (rewriter).
232725a3e95 Row pattern recognition patch (parse/analysis).
c71940f9e21 Row pattern recognition patch for raw parser.
322bab79744 Move declarations related to locktags from lock.h to new locktag.h

> In particular, could you also check whether the
> recent master commit titled "Fix integer overflow in
> nodeWindowAgg.c" (backpatched to 14) is included in your base?  It
> touches the same ROWS/GROUPS frame boundary code path that RPR
> interacts with, so its presence or absence will help narrow things
> down.  Once I can match the same base locally, I will reproduce and
> include the fix in the next revision.

No, the commit is not in my working tree.

So I guess I should have rebased v46 tree so that the commit is in the
work tree before applying your patches. Am I correct?

> A quick status update:
> 
>   - nocfbot-0006: self-review complete, not yet sent (comment tweak per
> review)
>   - nocfbot-0007: under self-review (comment details per review)
>   - nocfbot-0008: in progress
> 
> I would like to resolve the crash first before sending the next
> revision, so that the full set goes out together once our bases are
> aligned and A3 passes on your side.

Let me check the crash first. Since apparently the crash was caused by
my mis operation.

> One process question: so far I have been folding review feedback
> into the relevant existing patches (i.e. amending nocfbot-0008 to
> reflect your comments on it).

I prefer this way.

> Would you prefer that I instead
> append the review fixes as new patches at the end of the series
> (e.g. nocfbot-0032, 0033, ...), so the history of reviewed changes
> stays visible?  Happy to follow whichever workflow you find easier
> to review.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  306. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-04-17T14:24:37Z

    Hi Tatsuo,

You are right. I checked and found the 0006 patch I applied was wrong
> one.
>
> Sorry for confusion.
>

No worries. Having 31 patches in the series is bound to cause
confusion — that is on me for sending such a large set at once.

t-ishii$ git log --oneline -10|cat
> 6dd0765463a Row pattern recognition patch (typedefs.list).
> c71940f9e21 Row pattern recognition patch for raw parser.
> 322bab79744 Move declarations related to locktags from lock.h to new
> locktag.h
>

Thank you for the log. I can see that your base is the original v46
application, while mine is the cfbot rebase from April 1
(732acf9b7c6).

No, the commit is not in my working tree.
>
> So I guess I should have rebased v46 tree so that the commit is in the
> work tree before applying your patches. Am I correct?
>

Since the root cause was the patch file mix-up, a rebase would not
have been necessary in this case. That said, the related fix that
went into master ("Fix integer overflow in nodeWindowAgg.c") touches
the same file as several of my patches, so it may cause conflicts
when applying v47. Please let me know if anything is difficult to
resolve.

Let me check the crash first. Since apparently the crash was caused by
> my mis operation.
>

Once confirmed, I will send the corrected patches first.

Regarding the README.rpr suggestion from the 0008 review: the
documentation in execRPR.c has dependencies spread across the patch
series, so separating it mid-review would be disruptive. I plan to
split it out as part of the final patch list once all 31 patches have
been reviewed.

I prefer this way.


Good — I will continue folding review feedback into the relevant
existing patches. Patch numbers and subjects will stay stable across
revisions.

Regards,
Henson

  307. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-04-19T10:18:01Z

    >> So I guess I should have rebased v46 tree so that the commit is in the
>> work tree before applying your patches. Am I correct?
>>
> 
> Since the root cause was the patch file mix-up, a rebase would not
> have been necessary in this case. That said, the related fix that
> went into master ("Fix integer overflow in nodeWindowAgg.c") touches
> the same file as several of my patches, so it may cause conflicts
> when applying v47. Please let me know if anything is difficult to
> resolve.

Ok, I will not rebase current v46 and continue to apply your
incremental patches and review them, until we agree to ship v47.  I
may see conflicts while creating v47 patch sets. I will ask help if if
your assistance needed. Thanks in advance.

> Let me check the crash first. Since apparently the crash was caused by
>> my mis operation.
>>
> 
> Once confirmed, I will send the corrected patches first.

I confirmed 0001-0008 applied cleanly and see no compile
warning. Regression test passed, no crash.

> Regarding the README.rpr suggestion from the 0008 review: the
> documentation in execRPR.c has dependencies spread across the patch
> series, so separating it mid-review would be disruptive. I plan to
> split it out as part of the final patch list once all 31 patches have
> been reviewed.

Ok.

> I prefer this way.
> 
> 
> Good ― I will continue folding review feedback into the relevant
> existing patches. Patch numbers and subjects will stay stable across
> revisions.

Looking forward to seeing revised incremental patch sets.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  308. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-04-19T13:52:26Z

     Hi Tatsuo,

Thank you for the careful review and for confirming the build on
your side.

Ok, I will not rebase current v46 and continue to apply your
> incremental patches and review them, until we agree to ship v47.  I
> may see conflicts while creating v47 patch sets. I will ask help if if
> your assistance needed. Thanks in advance.
>

Understood.  Please let me know whenever a conflict comes up while
assembling v47 and I will help resolve it on my side.


> I confirmed 0001-0008 applied cleanly and see no compile
> warning. Regression test passed, no crash.
>

Thank you for confirming.


> > Regarding the README.rpr suggestion from the 0008 review: the
> > documentation in execRPR.c has dependencies spread across the patch
> > series, so separating it mid-review would be disruptive. I plan to
> > split it out as part of the final patch list once all 31 patches have
> > been reviewed.
>
> Ok.


Thank you.  I will prepare the README.rpr split as a follow-up patch
after the current 0031 once the series review is complete.

Looking forward to seeing revised incremental patch sets.


Attached is the revised v47 series, with the review comments on
0006, 0007, and 0008 incorporated.  Numbering and subjects are
unchanged from the previous send.

Since the edits in 0006-0008 propagate into the later patches --
mostly as context-line shifts, and in 0017/0023 as updates to the
rpr_integration expected output -- I have attached the full
0001-0031 set rather than only the updated patches.  Applying this
set on top of v46 should give the same final tree as before, just
with the review adjustments folded in.

Summary of changes from the previous v47:

  - 0006 (Fix DEFINE expression handling): per your comment, the
    first occurrence of "RPR" now carries the "(Row Pattern
    Recognition)" expansion and the later occurrence is the bare
    abbreviation.

  - 0007 (Add RPR planner integration tests): overall
    strengthening of the tests and their comments.  Each block
    now carries an intent comment describing what the block is
    trying to cover and what the expected plan or result is,
    the "Non-RPR / RPR" comment pair is reworded as complete
    sentences, and B8 is reworked so that the EXPLAIN plan
    actually exercises Incremental Sort.

    Because some blocks cover features that are only
    introduced later in the series, a few tests in 0007
    temporarily produce error output.  Concretely, B4
    (RPR + prepared statements) uses the 2-argument form
    PREV(val, $1), which is not yet available at the 0007
    point: it is introduced in 0017 (1-slot PREV/NEXT
    navigation), and the accompanying "Nav Mark Lookback"
    EXPLAIN property referenced in the B4 comments is added
    in 0023.  The rpr_integration expected-output file is
    updated by 0017 (to clear the PREV/prepared-statement
    errors) and then again by 0023 (to add the
    "Nav Mark Lookback" lines to the EXPLAIN output), so that
    by the end of the series the output is clean.

    XXX notes have been added in three places where a test
    records behaviour that may be revisited later:

      (1) The subquery-output test documents why the column
          guard in allpaths.c has to be conservative: a column
          referenced only by DEFINE is indistinguishable from
          an exposed column at the targetlist level, so
          remove_unused_subquery_outputs() cannot apply its
          NULL-replacement selectively without risking DEFINE
          evaluation on NULL inputs.  The XXX flags this as a
          candidate for a precise follow-up optimization.

      (2) The B7 Recursive CTE test carries an XXX noting that
          it is unclear whether this case falls under the
          ISO/IEC 9075-2 4.18.5 / 6.17.5 prohibition, and even
          if it does not, whether a query that does trigger
          the prohibition can be constructed at all.  The
          disposition is left to the community.

      (3) The B9 volatile-in-DEFINE test records current
          behaviour using "random() >= 0.0" (structurally
          equivalent to TRUE, so the expected output stays
          deterministic).  The XXX flags that if we decide to
          reject volatile functions in DEFINE, this test must
          be converted into an error-case test.  See the
          dedicated section at the end of this mail.

  - 0008 (Replace reduced frame map with single match result):
      * Comment block X-1 now uses the full field names
        rpr_match_valid / rpr_match_matched / rpr_match_length.
      * get_reduced_frame_status() sentence rephrased in the
        passive voice, as you suggested:
          "Row's status against the current match result can be
           obtained by calling get_reduced_frame_status()."
      * README.rpr split deferred per the above.

  - 0009-0031: no semantic code changes.  Most patches only
    carry context-line shifts from the edits above.  The two
    exceptions are 0017 and 0023, whose rpr_integration.out
    hunks are updated to match the revised B4 test in 0007
    (clearing the temporary errors and adding the
    "Nav Mark Lookback" lines, respectively).

On volatile functions in DEFINE (your 2026-04-13 mail): the SQL
standard itself is relatively narrow here -- the only explicit
restriction it places is that a navigation function's offset must
be a runtime constant.  Volatile expressions in the DEFINE
predicate are not forbidden by the standard.

That said, once we move into the pattern-matching machinery, the
observable semantics of a volatile DEFINE become quite murky.
Depending on the situation, the DEFINE predicate may be evaluated
once per row, or separately per active matching context, or some
combination of both as the NFA explores alternatives.  A user
writing "random() > 0.5" in DEFINE would have a hard time
predicting, let alone relying on, the evaluation count -- and I
have not been able to think of a realistic use case that actually
benefits from this flexibility.

Given the gap between "permitted by the standard" and "meaningful
to the user," my own leaning is toward prohibiting volatile
functions in DEFINE at transformation time, and -- if we go that
way -- also examining whether the new check would make the
existing runtime-constant check on the navigation-function offset
redundant, or whether the two could be folded into a single check
pass.  That said, this is ultimately a policy call, and I would rather defer
the final decision to your judgment as a senior member of the
community: does prohibition seem like the right direction, or
would you prefer to keep the current behaviour in place?

On sequencing, if we do take it up, I would suggest handling it
after the 31-patch set, alongside the README.rpr split as
follow-up work on top of 0031.  Whether it ultimately lands
inside v47 or as a separate piece on top does not need to be
decided right now -- there is still room to discuss it as the
review progresses, and I am happy to adjust either way based on
your direction.

Regards,
Henson

  309. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-04-27T04:52:43Z

    [Chao Li <li.evan.chao@gmail.com> is added to Cc: I accidentally
forgot to add him. Sorry, Chao.]

Hi Henson,

I looked through 0001-0031 patches.

>  Hi Tatsuo,
> 
> Thank you for the careful review and for confirming the build on
> your side.
> 
> Ok, I will not rebase current v46 and continue to apply your
>> incremental patches and review them, until we agree to ship v47.  I
>> may see conflicts while creating v47 patch sets. I will ask help if if
>> your assistance needed. Thanks in advance.
>>
> 
> Understood.  Please let me know whenever a conflict comes up while
> assembling v47 and I will help resolve it on my side.
> 
> 
>> I confirmed 0001-0008 applied cleanly and see no compile
>> warning. Regression test passed, no crash.
>>
> 
> Thank you for confirming.
> 
> 
>> > Regarding the README.rpr suggestion from the 0008 review: the
>> > documentation in execRPR.c has dependencies spread across the patch
>> > series, so separating it mid-review would be disruptive. I plan to
>> > split it out as part of the final patch list once all 31 patches have
>> > been reviewed.
>>
>> Ok.
> 
> 
> Thank you.  I will prepare the README.rpr split as a follow-up patch
> after the current 0031 once the series review is complete.
> 
> Looking forward to seeing revised incremental patch sets.
> 
> 
> Attached is the revised v47 series, with the review comments on
> 0006, 0007, and 0008 incorporated.  Numbering and subjects are
> unchanged from the previous send.

0001 was already reviewed (LGTM).
0002-0003 looks good to me.
0004 was already reviewed (LGTM).
0005 thanks for review.

> Since the edits in 0006-0008 propagate into the later patches --
> mostly as context-line shifts, and in 0017/0023 as updates to the
> rpr_integration expected output -- I have attached the full
> 0001-0031 set rather than only the updated patches.  Applying this
> set on top of v46 should give the same final tree as before, just
> with the review adjustments folded in.

Thank you for taking care of my request.

> Summary of changes from the previous v47:
> 
>   - 0006 (Fix DEFINE expression handling): per your comment, the
>     first occurrence of "RPR" now carries the "(Row Pattern
>     Recognition)" expansion and the later occurrence is the bare
>     abbreviation.

Confirmed. 0006 Looks good to me.

>   - 0007 (Add RPR planner integration tests): overall
>     strengthening of the tests and their comments.  Each block
>     now carries an intent comment describing what the block is
>     trying to cover and what the expected plan or result is,
>     the "Non-RPR / RPR" comment pair is reworded as complete
>     sentences, and B8 is reworked so that the EXPLAIN plan
>     actually exercises Incremental Sort.
> 
>     Because some blocks cover features that are only
>     introduced later in the series, a few tests in 0007
>     temporarily produce error output.  Concretely, B4
>     (RPR + prepared statements) uses the 2-argument form
>     PREV(val, $1), which is not yet available at the 0007
>     point: it is introduced in 0017 (1-slot PREV/NEXT
>     navigation), and the accompanying "Nav Mark Lookback"
>     EXPLAIN property referenced in the B4 comments is added
>     in 0023.  The rpr_integration expected-output file is
>     updated by 0017 (to clear the PREV/prepared-statement
>     errors) and then again by 0023 (to add the
>     "Nav Mark Lookback" lines to the EXPLAIN output), so that
>     by the end of the series the output is clean.
> 
>     XXX notes have been added in three places where a test
>     records behaviour that may be revisited later:
> 
>       (1) The subquery-output test documents why the column
>           guard in allpaths.c has to be conservative: a column
>           referenced only by DEFINE is indistinguishable from
>           an exposed column at the targetlist level, so
>           remove_unused_subquery_outputs() cannot apply its
>           NULL-replacement selectively without risking DEFINE
>           evaluation on NULL inputs.  The XXX flags this as a
>           candidate for a precise follow-up optimization.
> 
>       (2) The B7 Recursive CTE test carries an XXX noting that
>           it is unclear whether this case falls under the
>           ISO/IEC 9075-2 4.18.5 / 6.17.5 prohibition, and even
>           if it does not, whether a query that does trigger
>           the prohibition can be constructed at all.  The
>           disposition is left to the community.

Ok, let's discuss on this later on.

>       (3) The B9 volatile-in-DEFINE test records current
>           behaviour using "random() >= 0.0" (structurally
>           equivalent to TRUE, so the expected output stays
>           deterministic).  The XXX flags that if we decide to
>           reject volatile functions in DEFINE, this test must
>           be converted into an error-case test.  See the
>           dedicated section at the end of this mail.

Ok.

Comment to A1:

+-- A1. Frame optimization bypass
:
:
+-- Non-RPR window with default frame -> frame optimization applied
+EXPLAIN (COSTS OFF)
+SELECT count(*) OVER w FROM rpr_integ
+WINDOW w AS (ORDER BY id);
+            QUERY PLAN             
+-----------------------------------
+ WindowAgg
+   Window: w AS (ORDER BY id)
+   ->  Sort
+         Sort Key: id
+         ->  Seq Scan on rpr_integ
+(5 rows)

The comment says frame optimization applied. But I don't see any frame
option changes (see how the first query changes the frame options).

I thought a frame optimization is something like this.
EXPLAIN(COSTS OFF)
SELECT row_number() OVER W FROM generate_series(1,5) s(i)
WINDOW w AS (
       ORDER BY i
       ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
);
                      QUERY PLAN                      
------------------------------------------------------
 WindowAgg
   Window: w AS (ORDER BY i ROWS UNBOUNDED PRECEDING)
   ->  Sort
         Sort Key: i
         ->  Function Scan on generate_series s
(5 rows)

EXPLAIN(COSTS OFF)
SELECT row_number() OVER W FROM generate_series(1,5) s(i)
WINDOW w AS (
       ORDER BY i
       ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
       PATTERN (A+)
       DEFINE A AS i > 2
);
                                  QUERY PLAN                                  
------------------------------------------------------------------------------
 WindowAgg
   Window: w AS (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
   Pattern: a+"
   Nav Mark Lookback: 0
   ->  Sort
         Sort Key: i
         ->  Function Scan on generate_series s
(7 rows)

>   - 0008 (Replace reduced frame map with single match result):
>       * Comment block X-1 now uses the full field names
>         rpr_match_valid / rpr_match_matched / rpr_match_length.
>       * get_reduced_frame_status() sentence rephrased in the
>         passive voice, as you suggested:
>           "Row's status against the current match result can be
>            obtained by calling get_reduced_frame_status()."
>       * README.rpr split deferred per the above.

Thanks for the new patch. 0008 looks good to me.

>   - 0009-0031: no semantic code changes.  Most patches only
>     carry context-line shifts from the edits above.  The two
>     exceptions are 0017 and 0023, whose rpr_integration.out
>     hunks are updated to match the revised B4 test in 0007
>     (clearing the temporary errors and adding the
>     "Nav Mark Lookback" lines, respectively).

Comment to 0009:

+	/* Mark VAR in visited before duplicate check to prevent DFS loops */
+	winstate->nfaVisitedElems[WORDNUM(state->elemIdx)] |=
+		((bitmapword) 1 << BITNUM(state->elemIdx));

Why do not use bms here? Maybe winstate->nfaVisitedElems could become
quite big?

0010-0016 look good to me.

Comment to 0017:
+
+/*
+ * check_rpr_nav_expr
+ *		Validate a single RPRNavExpr node by walking its arg and offset_arg
+ *		subtrees in a single pass each.  Checks for nested PREV/NEXT, missing
+ *		column references, and non-constant offset expressions.
+ */

"Checks for" --> "Check for" (no "s").

+		ereport(ERROR,
+				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+				 errmsg("PREV and NEXT cannot be nested"),
+				 parser_errposition(pstate, nav->location)));

 errmsg("PREV and NEXT cannot be nested") ->
 errmsg("cannot nest PREV and NEXT") (avoid passive voice in error messages)

+	ereport(ERROR,
+			(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+			 errmsg("next() can only be used in a DEFINE clause")));

errmsg("next() can only be used in a DEFINE clause") ->
errmsg("cannot use next() other than in a DEFINE clause")

The same reason above. There are some places where similar changing
necessary.

+		if (offset > maxOffset)
+			maxOffset = offset;

+			if (offset > *maxOffset)
+				*maxOffset = offset;

You can use Max macro here.

0018-0023 look good to me.

Comment to 0024:

+        <function>first</function> ( <parameter>value</parameter> <type>anyelement</type> [, <parameter>offset</parameter> <type>bigint</type> ] )
+        <returnvalue>anyelement</returnvalue>
+       </para>
+       <para>
+        Returns the column value at the row <parameter>offset</parameter>
+        rows after the match start row;
+        returns NULL if the target row is beyond the current row.
+        <parameter>offset</parameter> defaults to 0 if omitted, referring to the
+        match start row itself.
+        <parameter>offset</parameter> must be a non-negative integer.
+        <parameter>offset</parameter> must not be NULL.
+        Can only be used in a <literal>DEFINE</literal> clause.
+       </para></entry>

"Returns the column value at the row" is not appropriate because
first() accepts any expression as the first argument. e.g. first(col1
+ col2).  Instead what about "Returns value evaluated at the row"?
Same thing can be said to other row pattern navigation operations.
I think they inherits my mistakes in my earlier patch. Sorry for this.

--- a/src/test/regress/expected/rpr.out
+++ b/src/test/regress/expected/rpr.out

+-- match_start=3(30): A=id3, B=id4, FIRST(val)=30≠10 → no match

Non ASCII letter(≠) should not be used.

0025-0031 look good to me.

> On volatile functions in DEFINE (your 2026-04-13 mail): the SQL
> standard itself is relatively narrow here -- the only explicit
> restriction it places is that a navigation function's offset must
> be a runtime constant.  Volatile expressions in the DEFINE
> predicate are not forbidden by the standard.
> 
> That said, once we move into the pattern-matching machinery, the
> observable semantics of a volatile DEFINE become quite murky.
> Depending on the situation, the DEFINE predicate may be evaluated
> once per row, or separately per active matching context, or some
> combination of both as the NFA explores alternatives.  A user
> writing "random() > 0.5" in DEFINE would have a hard time
> predicting, let alone relying on, the evaluation count -- and I
> have not been able to think of a realistic use case that actually
> benefits from this flexibility.

Agreed. The flexibility does not bring benefits to users.

> Given the gap between "permitted by the standard" and "meaningful
> to the user," my own leaning is toward prohibiting volatile
> functions in DEFINE at transformation time, and -- if we go that
> way -- also examining whether the new check would make the
> existing runtime-constant check on the navigation-function offset
> redundant, or whether the two could be folded into a single check
> pass.  That said, this is ultimately a policy call, and I would rather defer
> the final decision to your judgment as a senior member of the
> community: does prohibition seem like the right direction, or
> would you prefer to keep the current behaviour in place?

Let's prohibit volatile functions in DEFINE. Although this needs extra
checking, it would make proceeding codes and tests simpler.  If
others, especially Vik and Jacob, think differently, please join the
discussion.

> On sequencing, if we do take it up, I would suggest handling it
> after the 31-patch set, alongside the README.rpr split as
> follow-up work on top of 0031.  Whether it ultimately lands
> inside v47 or as a separate piece on top does not need to be
> decided right now -- there is still room to discuss it as the
> review progresses, and I am happy to adjust either way based on
> your direction.

I too prefer to have it after the 31-patch set.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  310. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-04-27T08:03:26Z

    Hi Tatsuo,

Thank you very much for the careful review covering 0001-0031 in a
single pass.

0001 was already reviewed (LGTM).
> 0002-0003 looks good to me.
> 0004 was already reviewed (LGTM).
> 0005 thanks for review.
>

Thank you for confirming.


> Comment to A1:
>
> +-- A1. Frame optimization bypass
> :
> :
> +-- Non-RPR window with default frame -> frame optimization applied
> +EXPLAIN (COSTS OFF)
> +SELECT count(*) OVER w FROM rpr_integ
> +WINDOW w AS (ORDER BY id);
> +            QUERY PLAN
> +-----------------------------------
> + WindowAgg
> +   Window: w AS (ORDER BY id)
> +   ->  Sort
> +         Sort Key: id
> +         ->  Seq Scan on rpr_integ
> +(5 rows)
>
> The comment says frame optimization applied. But I don't see any frame
> option changes (see how the first query changes the frame options).
>
> I thought a frame optimization is something like this.
> EXPLAIN(COSTS OFF)
> SELECT row_number() OVER W FROM generate_series(1,5) s(i)
> WINDOW w AS (
>        ORDER BY i
>        ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
> );
>                       QUERY PLAN
> ------------------------------------------------------
>  WindowAgg
>    Window: w AS (ORDER BY i ROWS UNBOUNDED PRECEDING)
>    ->  Sort
>          Sort Key: i
>          ->  Function Scan on generate_series s
> (5 rows)
>
> EXPLAIN(COSTS OFF)
> SELECT row_number() OVER W FROM generate_series(1,5) s(i)
> WINDOW w AS (
>        ORDER BY i
>        ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>        PATTERN (A+)
>        DEFINE A AS i > 2
> );
>                                   QUERY PLAN
>
>
> ------------------------------------------------------------------------------
>  WindowAgg
>    Window: w AS (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED
> FOLLOWING)
>    Pattern: a+"
>    Nav Mark Lookback: 0
>    ->  Sort
>          Sort Key: i
>          ->  Function Scan on generate_series s
> (7 rows)
>

Thank you for pointing this out.  I will analyse A1 and reflect
the findings in the next revision.


> Comment to 0009:
>
> +       /* Mark VAR in visited before duplicate check to prevent DFS loops
> */
> +       winstate->nfaVisitedElems[WORDNUM(state->elemIdx)] |=
> +               ((bitmapword) 1 << BITNUM(state->elemIdx));
>
> Why do not use bms here? Maybe winstate->nfaVisitedElems could become
> quite big?
>

Thank you for the suggestion.  I was not aware of Bitmapset when
this was written; I will switch nfaVisitedElems to Bitmapset.

On sizing, nfa->nelems is structurally bounded by RPR_ELEMIDX_MAX
(INT16_MAX), since RPRNFAState.elemIdx is int16 and scanRPRPattern()
rejects anything past that with "pattern too complex".  So the
bitmap is at most 4 KB per WindowAgg even in the worst case.


> 0010-0016 look good to me.
>

Confirmed.


> Comment to 0017:
> +
> +/*
> + * check_rpr_nav_expr
> + *             Validate a single RPRNavExpr node by walking its arg and
> offset_arg
> + *             subtrees in a single pass each.  Checks for nested
> PREV/NEXT, missing
> + *             column references, and non-constant offset expressions.
> + */
>
> "Checks for" --> "Check for" (no "s").
>

Thank you, will fix.


> +               ereport(ERROR,
> +                               (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
> +                                errmsg("PREV and NEXT cannot be nested"),
> +                                parser_errposition(pstate,
> nav->location)));
>
>  errmsg("PREV and NEXT cannot be nested") ->
>  errmsg("cannot nest PREV and NEXT") (avoid passive voice in error
> messages)
>
> +       ereport(ERROR,
> +                       (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
> +                        errmsg("next() can only be used in a DEFINE
> clause")));
>
> errmsg("next() can only be used in a DEFINE clause") ->
> errmsg("cannot use next() other than in a DEFINE clause")
>
> The same reason above. There are some places where similar changing
> necessary.
>

Thank you for the guidance.  I will fix the two messages called out
here and also look over the surrounding RPR errmsg() strings for
similar passive-voice phrasings, and rewrite those in the active
voice consistent with the PostgreSQL message style guide.


> +               if (offset > maxOffset)
> +                       maxOffset = offset;
>
> +                       if (offset > *maxOffset)
> +                               *maxOffset = offset;
>
> You can use Max macro here.
>

Thank you, will switch both sites to Max().

0018-0023 look good to me.
>

Confirmed.


> Comment to 0024:
>
> +        <function>first</function> ( <parameter>value</parameter>
> <type>anyelement</type> [, <parameter>offset</parameter>
> <type>bigint</type> ] )
> +        <returnvalue>anyelement</returnvalue>
> +       </para>
> +       <para>
> +        Returns the column value at the row <parameter>offset</parameter>
> +        rows after the match start row;
> +        returns NULL if the target row is beyond the current row.
> +        <parameter>offset</parameter> defaults to 0 if omitted, referring
> to the
> +        match start row itself.
> +        <parameter>offset</parameter> must be a non-negative integer.
> +        <parameter>offset</parameter> must not be NULL.
> +        Can only be used in a <literal>DEFINE</literal> clause.
> +       </para></entry>
>
> "Returns the column value at the row" is not appropriate because
> first() accepts any expression as the first argument. e.g. first(col1
> + col2).  Instead what about "Returns value evaluated at the row"?
> Same thing can be said to other row pattern navigation operations.
> I think they inherits my mistakes in my earlier patch. Sorry for this.
>

Thank you, and please do not apologise -- I should have caught this
when I carried the wording across.  I will reword the synopses for
first(), last(), prev(), next() (and any other entry that inherited
the same phrasing) along the line you suggested.


> --- a/src/test/regress/expected/rpr.out
> +++ b/src/test/regress/expected/rpr.out
>
> +-- match_start=3(30): A=id3, B=id4, FIRST(val)=30≠10 → no match
>
> Non ASCII letter(≠) should not be used.
>

Thank you, will fix and sweep rpr*.out for any other non-ASCII
characters (the → on the same line is also non-ASCII).


> 0025-0031 look good to me.
>

Confirmed.


> > On volatile functions in DEFINE (your 2026-04-13 mail): the SQL
> > standard itself is relatively narrow here -- the only explicit
> > restriction it places is that a navigation function's offset must
> > be a runtime constant.  Volatile expressions in the DEFINE
> > predicate are not forbidden by the standard.
> >
> > That said, once we move into the pattern-matching machinery, the
> > observable semantics of a volatile DEFINE become quite murky.
> > Depending on the situation, the DEFINE predicate may be evaluated
> > once per row, or separately per active matching context, or some
> > combination of both as the NFA explores alternatives.  A user
> > writing "random() > 0.5" in DEFINE would have a hard time
> > predicting, let alone relying on, the evaluation count -- and I
> > have not been able to think of a realistic use case that actually
> > benefits from this flexibility.
>
> Agreed. The flexibility does not bring benefits to users.
>
> > Given the gap between "permitted by the standard" and "meaningful
> > to the user," my own leaning is toward prohibiting volatile
> > functions in DEFINE at transformation time, and -- if we go that
> > way -- also examining whether the new check would make the
> > existing runtime-constant check on the navigation-function offset
> > redundant, or whether the two could be folded into a single check
> > pass.  That said, this is ultimately a policy call, and I would rather
> defer
> > the final decision to your judgment as a senior member of the
> > community: does prohibition seem like the right direction, or
> > would you prefer to keep the current behaviour in place?
>
> Let's prohibit volatile functions in DEFINE. Although this needs extra
> checking, it would make proceeding codes and tests simpler.  If
> others, especially Vik and Jacob, think differently, please join the
> discussion.
>

Thank you for the decision.  Since this is a policy change that may
still attract further community discussion, I would like to handle
it as a lower-priority, self-contained follow-up on top of 0031, so
it can be picked up, held, or revised independently.  Tentative
plan: reject volatile functions in DEFINE during parse-analysis via
contain_volatile_functions(), fold with the existing offset
runtime-constant check where the two share traversal, and -- if the
prohibition lands -- convert the B9 test into an error-case test
and drop the XXX note.


> > On sequencing, if we do take it up, I would suggest handling it
> > after the 31-patch set, alongside the README.rpr split as
> > follow-up work on top of 0031.  Whether it ultimately lands
> > inside v47 or as a separate piece on top does not need to be
> > decided right now -- there is still room to discuss it as the
> > review progresses, and I am happy to adjust either way based on
> > your direction.
>
> I too prefer to have it after the 31-patch set.
>

Thank you, confirmed.  Both items (volatile-DEFINE prohibition and
README.rpr split) will land as follow-up patches on top of 0031,
after the current series is committed.

One quick question: for the 0007 / 0009 / 0017 / 0024 review fixes
summarised below, would additional follow-up patches on top of 0031
be preferable to refreshing the already-reviewed series?


For ease of tracking, here is a one-line summary per patch of the
changes I plan to fold into the next revision:

  0007: analyse A1 and, if the diagnosis holds, revise it so the
        frame optimization bypass is observable in EXPLAIN
        (adjusting the comment and expected output accordingly).
  0009: switch WindowAggState.nfaVisitedElems from a bitmapword[]
        to Bitmapset, replacing the manual WORDNUM/BITNUM sites
        with bms_add_member() / bms_is_member() / bms_free().
  0017: fix the "Checks for" -> "Check for" comment, rewrite the
        called-out passive-voice errmsg() strings (and the similar
        ones nearby) in the active voice, and use Max() at the two
        offset-update sites.
  0024: reword the navigation synopses (first/last/prev/next) to
        "Returns the value of <value> evaluated at the row ...",
        and replace the non-ASCII ≠ / → in rpr.out with != / ->
        (sweeping the rest of rpr*.out for other non-ASCII).

In addition, I plan to follow the current series with two new
patches on top of 0031 (final numbering deferred until they are
posted):

  new: README.rpr split -- extract the design notes currently
       inlined in execRPR.c into src/backend/executor/README.rpr,
       as previously discussed.
  new: Prohibit volatile functions in DEFINE -- separate,
       self-contained patch as described above; intended as a
       lower-priority follow-up that can be picked up, held, or
       revised independently of the main series depending on how
       the community discussion settles.

Separately, one item is deferred for later discussion rather than
folded into a patch:

  deferred: B7 Recursive CTE XXX in 0007 -- whether this case
            falls under ISO/IEC 9075-2 4.18.5 / 6.17.5 prohibition
            ("Ok, let's discuss on this later on.").  Noted, happy
            to pick this up whenever convenient.

Once you let me know your preference on the sequencing question
above, I will send the corresponding patches as soon as they are
ready, and post the two follow-ups separately so that they can be
reviewed and committed on top of the main series.

Thank you again for the thorough review.

Regards,
Henson

  311. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-04-27T08:42:20Z

    Hi Henson,

> Thank you for pointing this out.  I will analyse A1 and reflect
> the findings in the next revision.

Thanks in advance.

>> Comment to 0009:
>>
>> +       /* Mark VAR in visited before duplicate check to prevent DFS loops
>> */
>> +       winstate->nfaVisitedElems[WORDNUM(state->elemIdx)] |=
>> +               ((bitmapword) 1 << BITNUM(state->elemIdx));
>>
>> Why do not use bms here? Maybe winstate->nfaVisitedElems could become
>> quite big?
>>
> 
> Thank you for the suggestion.  I was not aware of Bitmapset when
> this was written; I will switch nfaVisitedElems to Bitmapset.
> 
> On sizing, nfa->nelems is structurally bounded by RPR_ELEMIDX_MAX
> (INT16_MAX), since RPRNFAState.elemIdx is int16 and scanRPRPattern()
> rejects anything past that with "pattern too complex".  So the
> bitmap is at most 4 KB per WindowAgg even in the worst case.

Hmm. From the comment from bitmapset.c:
 * A bitmap set can represent any set of nonnegative integers, although
 * it is mainly intended for sets where the maximum value is not large,
 * say at most a few hundred.

Maybe 4 KB is tool large? If so, there's no need to rush to change the
implementation. You can leave it as it is for now.

>> Comment to 0024:
>>
>> +        <function>first</function> ( <parameter>value</parameter>
>> <type>anyelement</type> [, <parameter>offset</parameter>
>> <type>bigint</type> ] )
>> +        <returnvalue>anyelement</returnvalue>
>> +       </para>
>> +       <para>
>> +        Returns the column value at the row <parameter>offset</parameter>
>> +        rows after the match start row;
>> +        returns NULL if the target row is beyond the current row.
>> +        <parameter>offset</parameter> defaults to 0 if omitted, referring
>> to the
>> +        match start row itself.
>> +        <parameter>offset</parameter> must be a non-negative integer.
>> +        <parameter>offset</parameter> must not be NULL.
>> +        Can only be used in a <literal>DEFINE</literal> clause.
>> +       </para></entry>
>>
>> "Returns the column value at the row" is not appropriate because
>> first() accepts any expression as the first argument. e.g. first(col1
>> + col2).  Instead what about "Returns value evaluated at the row"?
>> Same thing can be said to other row pattern navigation operations.
>> I think they inherits my mistakes in my earlier patch. Sorry for this.
>>
> 
> Thank you, and please do not apologise -- I should have caught this
> when I carried the wording across.  I will reword the synopses for
> first(), last(), prev(), next() (and any other entry that inherited
> the same phrasing) along the line you suggested.

Thank you.

>> Let's prohibit volatile functions in DEFINE. Although this needs extra
>> checking, it would make proceeding codes and tests simpler.  If
>> others, especially Vik and Jacob, think differently, please join the
>> discussion.
>>
> 
> Thank you for the decision.  Since this is a policy change that may
> still attract further community discussion, I would like to handle
> it as a lower-priority, self-contained follow-up on top of 0031, so
> it can be picked up, held, or revised independently.  Tentative
> plan: reject volatile functions in DEFINE during parse-analysis via
> contain_volatile_functions(), fold with the existing offset
> runtime-constant check where the two share traversal, and -- if the
> prohibition lands -- convert the B9 test into an error-case test
> and drop the XXX note.

Your tentative plan looks good to me.

>> > On sequencing, if we do take it up, I would suggest handling it
>> > after the 31-patch set, alongside the README.rpr split as
>> > follow-up work on top of 0031.  Whether it ultimately lands
>> > inside v47 or as a separate piece on top does not need to be
>> > decided right now -- there is still room to discuss it as the
>> > review progresses, and I am happy to adjust either way based on
>> > your direction.
>>
>> I too prefer to have it after the 31-patch set.
>>
> 
> Thank you, confirmed.  Both items (volatile-DEFINE prohibition and
> README.rpr split) will land as follow-up patches on top of 0031,
> after the current series is committed.
> 
> One quick question: for the 0007 / 0009 / 0017 / 0024 review fixes
> summarised below, would additional follow-up patches on top of 0031
> be preferable to refreshing the already-reviewed series?

Yes, that's fine. I will apply the delta patch for 0007 / 0009 / 0017
/ 0024 on top of 0031.

> For ease of tracking, here is a one-line summary per patch of the
> changes I plan to fold into the next revision:
> 
>   0007: analyse A1 and, if the diagnosis holds, revise it so the
>         frame optimization bypass is observable in EXPLAIN
>         (adjusting the comment and expected output accordingly).
>   0009: switch WindowAggState.nfaVisitedElems from a bitmapword[]
>         to Bitmapset, replacing the manual WORDNUM/BITNUM sites
>         with bms_add_member() / bms_is_member() / bms_free().
>   0017: fix the "Checks for" -> "Check for" comment, rewrite the
>         called-out passive-voice errmsg() strings (and the similar
>         ones nearby) in the active voice, and use Max() at the two
>         offset-update sites.
>   0024: reword the navigation synopses (first/last/prev/next) to
>         "Returns the value of <value> evaluated at the row ...",
>         and replace the non-ASCII ≠ / → in rpr.out with != / ->
>         (sweeping the rest of rpr*.out for other non-ASCII).

Confirmed.

> In addition, I plan to follow the current series with two new
> patches on top of 0031 (final numbering deferred until they are
> posted):
> 
>   new: README.rpr split -- extract the design notes currently
>        inlined in execRPR.c into src/backend/executor/README.rpr,
>        as previously discussed.
>   new: Prohibit volatile functions in DEFINE -- separate,
>        self-contained patch as described above; intended as a
>        lower-priority follow-up that can be picked up, held, or
>        revised independently of the main series depending on how
>        the community discussion settles.

Ok.

> Separately, one item is deferred for later discussion rather than
> folded into a patch:
> 
>   deferred: B7 Recursive CTE XXX in 0007 -- whether this case
>             falls under ISO/IEC 9075-2 4.18.5 / 6.17.5 prohibition
>             ("Ok, let's discuss on this later on.").  Noted, happy
>             to pick this up whenever convenient.

Ok.

> Once you let me know your preference on the sequencing question
> above, I will send the corresponding patches as soon as they are
> ready, and post the two follow-ups separately so that they can be
> reviewed and committed on top of the main series.
> 
> Thank you again for the thorough review.

Thank you for your effort!
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  312. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-05-01T17:11:00Z

    Hi Tatsuo,


Attached are 40 incremental patches on top of v46, with all
review-driven follow-ups folded in.  The 0001-0031 portion is
unchanged from the previous send (filenames preserved).  The new
patches 0032-0040 are the delta + follow-up patches we agreed to
deliver on top of 0031, plus two cosmetic patches that surfaced
during the sweep.

As you suggested, I have not refreshed 0001-0031; the new work
is layered on top so that the existing review state on those
patches stays valid.

Numbering and subjects of 0001-0031 are unchanged.  The new
patches are:


  - 0032 Fix A1 frame optimization bypass test in
    rpr_integration

    Per your A1 review.  Switches the baseline to row_number()
    with an explicit ROWS frame so that the bypass guard is
    actually exercised.  Block comment reworded in A2/A3 style.


  - 0033 Fix imperative voice in check_rpr_nav_expr header
    comment

    Per your typo note: "Checks for" -> "Check for".


  - 0034 Fix passive voice in RPR error messages

    Per your style note.  Sweep of RPR-touched code paths
    (parse_rpr.c, parse_func.c, windowfuncs.c) for the
    "is not permitted when ... is used" / "can only be used in"
    patterns, rewritten to "cannot use ... with" /
    "cannot use ... outside".


  - 0035 Use Max()/Min() macros for RPR extremum accumulation

    Per your Max() macro suggestion.  Sweep of all hand-rolled
    "if (x > y) y = x;" sites in createplan.c, nodeWindowAgg.c,
    and execRPR.c.  No double-evaluation hazards on the call
    sites.


  - 0036 Reword RPR navigation function synopses

    Per your sgml note.  "Returns the column value at the row"
    -> "Returns value evaluated at the row" (the first argument
    can be an arbitrary expression).


  - 0037 Replace non-ASCII characters in RPR code and tests

    Per your non-ASCII note.  Sweep of RPR-touched files;
    mostly comments and rpr*.out.


  - 0038 Move RPR design notes from execRPR.c to README.rpr

    Per the README.rpr split we discussed.  Extracts the
    design block at the head of execRPR.c into
    src/backend/executor/README.rpr (plain text, matching the
    executor/README convention).  No code changes.


  - 0039 Apply pgindent canonicalization to RPR comment blocks

    Cosmetic.  pgindent sweep across RPR-touched files; labeled
    blocks wrapped in /*---------- ... *---------- to preserve
    hanging indents.  No code changes.


  - 0040 Reword RPR navigation function descriptions in
    pg_proc.dat

    Cosmetic.  Aligns the eight RPR navigation descr fields
    with the surrounding "fetch the ... row value" pattern used
    by lag / lead / first_value / last_value / nth_value.


On the items deferred to post-v47:


  - DEFINE volatile prohibition: prepared as a self-contained
    follow-up so it can be reviewed independently of v47.

  - 0009 nfaVisitedElems representation: per your decision,
    kept as-is.  Three points behind that:

      (1) Size concern.  visited is 1 bit per element, a flat
          NFA element is 16 bytes -- a fixed 128:1 ratio
          (~0.78%).  The bitmap is dwarfed by the NFA it
          tracks (worst case ~4 KiB next to ~512 KiB), so it
          is not a memory hot spot.

      (2) Bitmapset misfit.  Its invariants ("no trailing zero
          word", "empty set = NULL") map the per-advance reset
          onto pfree + palloc-on-next-add, which defeats the
          in-place / fixed-capacity reuse pattern this code
          relies on.

      (3) Future variant.  A high-water mark approach (track
          the min/max touched word index, memset only that
          span on reset) preserves the in-place pattern and
          shrinks reset cost from O(numElements/64) to
          O(span_words).  Held as a future optimization
          candidate.

  - 0007 / B7 (Recursive CTE XXX): tracked as a discussion item
    awaiting community input on the standard interpretation.


Regards,
Henson

  313. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-05-02T04:17:18Z

    Hi Henson,

Thank you for the patch sets! Quick question:

I know in v47 JIT compilation supports PREV/NEXT. What about
FIRST/LAST and compound forms?

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  314. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-05-02T04:30:25Z

    Hi Tatsuo,

I know in v47 JIT compilation supports PREV/NEXT. What about
> FIRST/LAST and compound forms?
>

Yes, those are JIT-compiled too in v47.

The JIT side only deals with two opcodes -- EEOP_RPR_NAV_SET
and EEOP_RPR_NAV_RESTORE.

FIRST/LAST and the compound forms differ only in how
ExecEvalRPRNavSet() computes target_pos for the slot swap;
the opcode interface and the call into ExecEvalRPRNavSet /
ExecEvalRPRNavRestore are unchanged.

So 0018's JIT support carries over to FIRST/LAST and
compounds without any JIT-side change.

Regards,
Henson

  315. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-05-02T04:38:28Z

    Hi Henson,

> I know in v47 JIT compilation supports PREV/NEXT. What about
>> FIRST/LAST and compound forms?
>>
> 
> Yes, those are JIT-compiled too in v47.
> 
> The JIT side only deals with two opcodes -- EEOP_RPR_NAV_SET
> and EEOP_RPR_NAV_RESTORE.
> 
> FIRST/LAST and the compound forms differ only in how
> ExecEvalRPRNavSet() computes target_pos for the slot swap;
> the opcode interface and the call into ExecEvalRPRNavSet /
> ExecEvalRPRNavRestore are unchanged.
> 
> So 0018's JIT support carries over to FIRST/LAST and
> compounds without any JIT-side change.

Ok, from user's point of view, PREV/NEXT/FIRST/LAST and compound forms
are all JIT-compiled. I will add it to the "major changes" section in
the v47 patches posting.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  316. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-05-02T05:03:04Z

    Attached is the v47 patches for Row pattern recognition (SQL/RPR).

Major changes from v46 include:

- Change implementation of row pattern navigation operations using
  "1-slot model", which allows to implement more standard compliant
  features such as an offset argument, more row pattern navigation
  operations (FIRST, LAST) and compound forms.

- Row pattern navigation operations now support FIRST, LAST and
  compound forms

- Add JIT compilation support for all row pattern navigation
  operations (including compound forms)

- Add tuplestore trim optimization for RPR PREV navigation

- Window function last_value() now allows to set mark in certain cases

- Change the implementation of reduced frame map. Now consumes less CPU and memory

- Add more optimization (absorption). e.g. (A B B)+

- Add planner integration tests (rpr_integration.sql)

- Add src/backend/executor/README.rpr (previously was in ExecRPR.c)

Current status:

The series of patches are to implement the row pattern recognition
(SQL/RPR) feature. Currently the implementation is a subset of SQL/RPR
(ISO/IEC 19075-2:2016). Namely, implementation of some features of
R020 (WINDOW clause). R010 (MATCH_RECOGNIZE) is out of the scope of
the patches.

Currently following features are implemented in the patches.

- PATTERN
- PATTERN regular expressions (+, *, ?)
  alternation (|), grouping () , {n}, {n,}, {n,m}, {,m}
  reluctant quantifiers (*? etc.),
- DEFINE
- INITIAL
- AFTER MATCH SKIP TO PAST LAST ROW
- AFTER MATCH SKIP TO NEXT ROW
- Row pattern navigation (FIRST, LAST, PREV, NEXT and their compound forms)

Currently following features are not implemented in the patches.

- MEASURES
- Pattern variable name qualified column reference (e.g. A.price)
- SUBSET
- SEEK
- AFTER MATCH SKIP TO
- AFTER MATCH SKIP TO FIRST
- AFTER MATCH SKIP TO LAST
- PATTERN regular expression  {- and -}, () (empty pattern)
  Anchors (^, $) are not permitted with RPR in Window clause by the
  standard.
- PERMUTE
- CLASSIFIER

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  317. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-05-03T05:34:47Z

    Hi Tatsuo, hi hackers,

I'd like to ask for a bit of review help before PGConf.dev 2026
(Vancouver, May 19–22).

For the SQL/RPR R020 poster going on display at the conference, I
have prepared a companion deep-dive page to sit behind the on-site
QR code, building on the patch series Tatsuo has been driving in
this thread:

  https://assam258-5892.github.io/postgresql/rpr/

It walks through the standard scope (R010 vs R020), worked examples,
NFA semantics, navigation functions, parser-to-executor design and
pruning rules, plus an interactive NFA simulator. To widen reach,
the page is also available in 23 AI-pretranslated languages from
the switcher:
العربية, Čeština, Dansk, Deutsch, Español, Suomi, Français,
עברית, Magyar, Italiano, 日本語, 한국어, Nederlands, Norsk,
Polski, Português (BR), Română, Русский, Slovenčina, Svenska,
Türkçe, 中文 (简体), 繁體中文.
Each translated page carries a clear "AI pre-translated, errors
possible" notice.

The ask (light, entirely optional):

  - English original has had one review pass; Korean I reviewed
    myself.
  - Tatsuo — if you ever find a free moment, a quick glance at the
    Japanese version would be very much appreciated, but absolutely
    no rush.
  - Native speakers of any of the other 22 languages — incidental
    corrections welcome (this thread or off-list), but nothing is
    expected.

References (also linked from the deep-dive page):
  - thread:
https://www.postgresql.org/message-id/flat/20230625.210509.1276733411677577841.t-ishii%40sranhm.sra.co.jp
  - latest patch (v47):
https://postgr.es/m/20260502.140304.670813149418899420.ishii@postgresql.org
  - commitfest #4460: https://commitfest.postgresql.org/patch/4460/

Best regards,
Henson

  318. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-05-05T00:01:24Z

    > Attached is the v47 patches for Row pattern recognition (SQL/RPR).

v47 starts to check whether range variable qualified expressions are
used in DEFINE clause. If used, raise an error. This is good because
we don't support the syntax yet (pattern variable range var), or it's
prohibited (from clause range var). However, the error message may not
be appropreate for the case when complex data type is involved.

CREATE TEMP TABLE item (name TEXT, amount INT);
CREATE TABLE
CREATE TEMP TABLE sales(items item);
CREATE TABLE
SELECT (items).name, (items).amount, count(*) OVER w
FROM sales WINDOW w AS (
    ORDER BY (items).name
    ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
    AFTER MATCH SKIP PAST LAST ROW
    PATTERN (A+)
    DEFINE A AS (A.items).amount > 10
);
ERROR:  pattern variable qualified column reference "a.items" is not supported in DEFINE clause
LINE 7:     DEFINE A AS (A.items).amount > 10
                         ^
If I change the DEFINE clause to:

    DEFINE A AS (sales.items).amount > 10

I get:

ERROR:  range variable qualified column reference "sales.items" is not allowed in DEFINE clause
LINE 8:     DEFINE A AS (sales.items).amount > 10
                         ^

In both cases, "a.items" or "sales.items" in the error messages are
not column names, therefore the wording "column reference" in the
error messages are not appropriate.

The checks are in transformColumnRef().  Its argument "ColumnRef
*cref->fields" is a list of column ref fields in DEFINE expression in
the raw parse tree.  Usually it's something like range_var.col_name or
just col_name etc. So it works. However when complex data types are
involved, cref->fields is something like sales.items. No col_name
included. In this case, ColumnRef is under Indirection node, and the
col_name is stored in Indirection->indirection, not in ColumnRef.
Therefore there's no way for transformColumnRef() to know the col_name.

In order to fix the issue, I think we need to add code to understand
range var qualification form "A.item" or "sales.items" in complex data
types case. But is it worth the trouble? The reword is just nicer
error messages. If we support MEASURES, the code is useful. But so far
we have decided to not support it in the first cut of RPR.

Maybe we should just change the error messages something like below
for now?

ERROR:  pattern variable qualified expression "a.items" is not supported in DEFINE clause
ERROR:  range variable qualified expression "sales.items" is not allowed in DEFINE clause

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  319. Re: Row pattern recognition

    jian he <jian.universality@gmail.com> — 2026-05-05T03:53:07Z

    On Tue, May 5, 2026 at 8:02 AM Tatsuo Ishii <ishii@postgresql.org> wrote:
>
> > Attached is the v47 patches for Row pattern recognition (SQL/RPR).
>

Hi
The patchset is too big, I only downloaded v47-0001...
Just a quick skim of the code on my end.
I noticed two code style issues, currently we don't have a code style
reference in the manual.

1. We need to add trailing commas to enum definitions. See
https://git.postgresql.org/cgit/postgresql.git/commit/?id=611806cd726fc92989ac918eac48fd8d684869c7

2.
+ | '{' ',' Iconst '}' Op
+ {
+ if (strcmp($5, "?") != 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("invalid token after range quantifier"),
+ errhint("Only \"?\" is allowed after {n,} or {,m} to make it reluctant."),
+ parser_errposition(@5)));
+ if ($3 <= 0 || $3 >= INT_MAX)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("quantifier bound must be between 1 and %d", INT_MAX - 1),
+ parser_errposition(@3));
+ $$ = (Node *) makeRPRQuantifier(0, $3, @5, @1, yyscanner);

+ (errcode(ERRCODE_SYNTAX_ERROR),
The leading parenthesis is optional and can be removed, fewer
parentheses are always better.
See related discussion:
https://postgr.es/m/202510100916.s2e6n3xiwvyc@alvherre.pgsql

Since v47-0001 introduces many ereport(ERROR) messages, it makes sense
to move the related regression tests from the other patch into
v47-0001, IMHO.



--
jian
https://www.enterprisedb.com/

  320. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-05-05T04:26:56Z

    Hi Tatsuo,

Thanks for the close look.  Inline below, then a sketch of what I
plan to put in v48 so you can pick what to pull first.

v47 starts to check whether range variable qualified expressions are
> used in DEFINE clause. If used, raise an error. This is good because
> we don't support the syntax yet (pattern variable range var), or it's
> prohibited (from clause range var). However, the error message may not
> be appropreate for the case when complex data type is involved.
>
> CREATE TEMP TABLE item (name TEXT, amount INT);
> CREATE TABLE
> CREATE TEMP TABLE sales(items item);
> CREATE TABLE
> SELECT (items).name, (items).amount, count(*) OVER w
> FROM sales WINDOW w AS (
>     ORDER BY (items).name
>     ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>     AFTER MATCH SKIP PAST LAST ROW
>     PATTERN (A+)
>     DEFINE A AS (A.items).amount > 10
> );
> ERROR:  pattern variable qualified column reference "a.items" is not
> supported in DEFINE clause
> LINE 7:     DEFINE A AS (A.items).amount > 10
>                          ^
> If I change the DEFINE clause to:
>
>     DEFINE A AS (sales.items).amount > 10
>
> I get:
>
> ERROR:  range variable qualified column reference "sales.items" is not
> allowed in DEFINE clause
> LINE 8:     DEFINE A AS (sales.items).amount > 10
>                          ^
>
> In both cases, "a.items" or "sales.items" in the error messages are
> not column names, therefore the wording "column reference" in the
> error messages are not appropriate.
>

Agreed.  Even for the simple non-composite cases the noun is fuzzy
once A_Indirection enters the picture.  "expression" reads naturally
in both cases.


> In order to fix the issue, I think we need to add code to understand
> range var qualification form "A.item" or "sales.items" in complex data
> types case. But is it worth the trouble? The reword is just nicer
> error messages. If we support MEASURES, the code is useful. But so far
> we have decided to not support it in the first cut of RPR.
>

Not now.  Teaching the pre-check about the surrounding A_Indirection
duplicates work that MEASURES will need anyway, and the only visible
gain before MEASURES is a more accurate echo of the source text.

Instead I'll mark the limitation in three places so future MEASURES
work can't miss it: a block comment above the pre-check that names
the limitation and the revisit point, an XXX cross-reference in
parse_rpr.c pointing back to the pre-check, and composite-type cases
in rpr_base whose expected output quotes only the ColumnRef portion
-- so when indirection-aware quoting lands, those outputs churn as
a tripwire.


> Maybe we should just change the error messages something like below
> for now?
>
> ERROR:  pattern variable qualified expression "a.items" is not supported
> in DEFINE clause
> ERROR:  range variable qualified expression "sales.items" is not allowed
> in DEFINE clause


Yes -- planned for v48.  While I'm in that path I'll also clean up
two adjacent issues:

  - the pre-check is binary, so an unknown qualifier (typo,
    undefined name) is misreported as a range-variable reference
    with SYNTAX_ERROR instead of falling through to the standard
    "column does not exist" / "missing FROM-clause entry"
    diagnostic;

  - parse_rpr.c and the SELECT docs claim DEFINE-name "collection"
    and "filtering during planning"; the actual behavior is
    validate-and-reject at parse analysis.


------------------------------------------------------------------
v48 follow-up plan
------------------------------------------------------------------

The items below are independent, so they can ship as separate
patches or as a single batched posting -- whichever you prefer.
Order is just for narrative; nothing depends on anything else.

  [A] Sizable refactor: collapse the four DEFINE walkers across
      parser/planner/executor into a single phase-tagged
      traversal.  On that base, reject volatile / NextValueExpr
      in DEFINE (the NFA may re-evaluate predicates during
      backtracking; STABLE / IMMUTABLE remain accepted), and
      bundle a STABLE/IMMUTABLE baseline test as a guard against
      accidental over-rejection.

  [B] Make the empty-match path observable in tests: replace the
      stale XXX comments in rpr_nfa with the actual behavior, and
      add EXPLAIN ANALYZE coverage in rpr_explain that surfaces
      "NFA: N matched (len 0/0/0.0)" so the NFA-found-but-window-
      empty case is regression-visible.

  [C] Trim the per-advance NFA visited-bitmap reset to a high-water
      mark range instead of the full bitmap.  Tradeoff: two int16
      comparisons added per visit, paying off for larger NFAs but
      added overhead for single-word bitmaps; semantics unchanged.
      I'll leave the decision to apply to your judgment.

  [D] DEFINE qualifier diagnostic: tri-classify (pattern var /
      range var / fall-through), reword to "expression", add
      unknown-qualifier and composite-type tests, and sync the
      adjacent stale comments and SELECT doc.

If any of [A]..[D] looks misjudged or you'd prefer a different
slicing, I'll reshape before posting.

Best,
Henson

  321. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-05-05T08:08:45Z

    Hi Jian,

> Hi
> The patchset is too big, I only downloaded v47-0001...

Yeah, since the feature is complex...

> 1. We need to add trailing commas to enum definitions. See
> https://git.postgresql.org/cgit/postgresql.git/commit/?id=611806cd726fc92989ac918eac48fd8d684869c7

I forgot that. Thanks for reminding.

> 2.
> + | '{' ',' Iconst '}' Op
> + {
> + if (strcmp($5, "?") != 0)
> + ereport(ERROR,
> + (errcode(ERRCODE_SYNTAX_ERROR),
> + errmsg("invalid token after range quantifier"),
> + errhint("Only \"?\" is allowed after {n,} or {,m} to make it reluctant."),
> + parser_errposition(@5)));
> + if ($3 <= 0 || $3 >= INT_MAX)
> + ereport(ERROR,
> + errcode(ERRCODE_SYNTAX_ERROR),
> + errmsg("quantifier bound must be between 1 and %d", INT_MAX - 1),
> + parser_errposition(@3));
> + $$ = (Node *) makeRPRQuantifier(0, $3, @5, @1, yyscanner);
> 
> + (errcode(ERRCODE_SYNTAX_ERROR),
> The leading parenthesis is optional and can be removed, fewer
> parentheses are always better.
> See related discussion:
> https://postgr.es/m/202510100916.s2e6n3xiwvyc@alvherre.pgsql

Right. I will review all other patches.

> Since v47-0001 introduces many ereport(ERROR) messages, it makes sense
> to move the related regression tests from the other patch into
> v47-0001, IMHO.

Thanks for the suggestion but I do not agree the direction.  It will
make the patch maintenance work harder. Also it will make it difficult
to understand the regression test organization.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  322. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-05-05T09:15:29Z

    Hi Tatsuo, Jian,

Both coding-style items from Jian's review will be addressed and
included in the next patch series.

> 1. We need to add trailing commas to enum definitions. See
> >
> https://git.postgresql.org/cgit/postgresql.git/commit/?id=611806cd726fc92989ac918eac48fd8d684869c7
>
> I forgot that. Thanks for reminding.
>

Three enum types introduced by the patch lack a trailing comma on the
last enumerator: RPRNavKind, RPRNavOffsetKind, and RPRPatternNodeType.
All three will be corrected in the next series.


> > + (errcode(ERRCODE_SYNTAX_ERROR),
> > The leading parenthesis is optional and can be removed, fewer
> > parentheses are always better.
> > See related discussion:
> > https://postgr.es/m/202510100916.s2e6n3xiwvyc@alvherre.pgsql
>
> Right. I will review all other patches.


The outer-parentheses pattern appears in 17 ereport() calls in
parse_rpr.c and 2 in optimizer/plan/rpr.c.  All 19 sites will be
cleaned up in the next series.

Jian, I will review your other patches as well.

Best,
Henson

  323. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-05-05T09:42:28Z

    Hi Henson,

> The outer-parentheses pattern appears in 17 ereport() calls in
> parse_rpr.c and 2 in optimizer/plan/rpr.c.  All 19 sites will be
> cleaned up in the next series.

I see them in gram.y as well. See attached patch.

> Jian, I will review your other patches as well.
> 
> Best,
> Henson

  324. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-05-05T09:59:12Z

    Hi Tatsuo,

I see them in gram.y as well. See attached patch.
>

Thank you for the patch. It will be included in the next series.

Best,
Henson

  325. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-05-07T12:41:46Z

    Hi Tatsuo, hi Jian,

Attached are 10 incremental patches on top of v47, covering the
follow-up plan ([A]..[D]) I sketched on 2026-05-05, Jian's two
coding-style items (trailing commas, outer parentheses; gram.y
folded into 0007), one comment-only patch (0009), and one trivial
cleanup (0010).  Packaging into the next version is at your
discretion.

Patch summary:

  - 0001 Add DEFINE non-volatile baseline to rpr_integration B9
      Adds a STABLE/IMMUTABLE baseline ahead of the volatile
      case so the rejection in 0002 is visible against a working
      reference.

  - 0002 Unify RPR DEFINE walkers and reject volatile callees
      [A].  Collapses the four DEFINE walkers (parser / planner
      / executor) into one phase-tagged traversal and, on that
      base, rejects volatile / NextValueExpr at parse-analysis.
      STABLE / IMMUTABLE accepted.  The offset runtime-constant
      check is folded in; B9 flips to error-case and its XXX is
      dropped.

  - 0003 Cover RPR empty-match path with EXPLAIN tests; fix
         stale XXX comments
      [B].  Adds rpr_explain coverage that surfaces "NFA: N
      matched (len 0/0/0.0)" so the NFA-found-but-empty-frame
      path is regression-visible; replaces stale rpr_nfa XXXs.

  - 0004 Reclassify DEFINE qualifier check and reword
         diagnostic to "expression"
      [D].  Tri-classifies (pattern var / range var / fall-
      through) so unknown qualifiers fall through to the
      standard "column does not exist" diagnostic.  Reworded to
      "expression" since the quoted token may include
      indirection on composite types (e.g. (A.items).amount).

  - 0005 Sync stale comments on DEFINE/PATTERN handling
      validateRPRPatternVarCount() rejects DEFINE names not in
      PATTERN, but comments / SELECT doc described it as
      "collection" / "filtered during planning".  Wording
      aligned; XXX note's "column references" bumped to
      "expressions" to match 0004.  Comment + doc only.

  - 0006 Add trailing commas to RPR enum definitions
      Per Jian.  RPRNavKind, RPRNavOffsetKind, RPRPatternNodeType.

  - 0007 Remove optional outer parentheses from ereport() calls
         in RPR files
      Per Jian + Tatsuo's gram.y patch.  19 sites in
      parse_rpr.c / optimizer/plan/rpr.c plus the gram.y sites.

  - 0008 Add high-water mark tracking to NFA visited bitmap
         reset
      [C].  Resets only the touched span (O(span_words) instead
      of O(numElements/64)), at the cost of two int16 comparisons
      per visit.  Semantics unchanged; happy to drop if you
      prefer the simpler bulk reset.

  - 0009 Document DEFINE subquery rejection as intentional
         over-rejection
      Comment-only.  Records that ISO/IEC 9075-2 §4.18.4 /
      6.17.4 (R010 / R020) permits a non-RPR, non-pattern-var-
      correlated subquery in DEFINE, and that our blanket
      rejection is deliberate over-rejection.  The case
      distinction (walk subquery Query for nested RPR; match
      ColumnRef qualifiers against ancestor p_rpr_pattern_vars)
      is doable with existing infrastructure and left as future
      work, not blocked on any other feature.

  - 0010 Remove duplicate #include in nodeWindowAgg.c
      "common/int.h" was included twice and the first occurrence
      was misplaced between catalog/* headers.  Drops the
      misplaced first; the second sits at the correct
      alphabetical position.

Still deferred:
  - B7 Recursive CTE XXX: pending community input on the
    §4.18.5 / 6.17.5 interpretation.

Please let me know if any of 0001-0010 looks misjudged or if you
would prefer a different slicing.

Best regards,
Henson

  326. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-05-09T07:43:18Z

    Hi Tatsuo,


Eleven incremental patches on top of v47.  Patches 0001-0010
are the same set I posted on 2026-05-07, with cosmetic touches
only.  Patch 0011 is new and tightens the consistency of how
this code base cites the SQL standard.

So far I have been pushing ahead narrowly on the feature
implementation itself.  I am aware I still have much to learn
about the PostgreSQL project's culture and conventions, and
your continued guidance would be very much appreciated.


Patch summary:


  - 0001 Add DEFINE non-volatile baseline to rpr_integration B9
      Adds a STABLE/IMMUTABLE baseline ahead of the volatile
      case so the rejection in 0002 is visible against a working
      reference.


  - 0002 Unify RPR DEFINE walkers and reject volatile callees
      [A].  Collapses the four DEFINE walkers (parser / planner
      / executor) into one phase-tagged traversal and, on that
      base, rejects volatile / NextValueExpr at parse-analysis.
      STABLE / IMMUTABLE accepted.  The offset runtime-constant
      check is folded in; B9 flips to error-case and its XXX is
      dropped.


  - 0003 Cover RPR empty-match path with EXPLAIN tests; fix
         stale XXX comments
      [B].  Adds rpr_explain coverage that surfaces "NFA: N
      matched (len 0/0/0.0)" so the NFA-found-but-empty-frame
      path is regression-visible; replaces stale rpr_nfa XXXs.


  - 0004 Reclassify DEFINE qualifier check and reword
         diagnostic to "expression"
      [D].  Tri-classifies (pattern var / range var / fall-
      through) so unknown qualifiers fall through to the
      standard "column does not exist" diagnostic.  Reworded to
      "expression" since the quoted token may include
      indirection on composite types (e.g. (A.items).amount).


  - 0005 Sync stale comments on DEFINE/PATTERN handling
      validateRPRPatternVarCount() rejects DEFINE names not in
      PATTERN, but comments / SELECT doc described it as
      "collection" / "filtered during planning".  Wording
      aligned; XXX note's "column references" bumped to
      "expressions" to match 0004.  Comment + doc only.


  - 0006 Add trailing commas to RPR enum definitions
      Per Jian.  RPRNavKind, RPRNavOffsetKind, RPRPatternNodeType.


  - 0007 Remove optional outer parentheses from ereport() calls
         in RPR files
      Per Jian + your gram.y patch.  19 sites in parse_rpr.c /
      optimizer/plan/rpr.c plus the gram.y sites.


  - 0008 Add high-water mark tracking to NFA visited bitmap
         reset
      [C].  Resets only the touched span (O(span_words) instead
      of O(numElements/64)), at the cost of two int16 comparisons
      per visit.  Semantics unchanged; happy to drop if you
      prefer the simpler bulk reset.


  - 0009 Document DEFINE subquery rejection as intentional
         over-rejection
      Comment-only.  Records that SQL/RPR permits a non-RPR,
      non-pattern-var-correlated subquery in DEFINE (see 0011
      for the normalized citation), and that our blanket
      rejection is deliberate over-rejection.  The case
      distinction (walk subquery Query for nested RPR; match
      ColumnRef qualifiers against ancestor p_rpr_pattern_vars)
      is doable with existing infrastructure and left as future
      work, not blocked on any other feature.


  - 0010 Remove duplicate #include in nodeWindowAgg.c
      "common/int.h" was included twice and the first occurrence
      was misplaced between catalog/* headers.  Drops the
      misplaced first; the second sits at the correct
      alphabetical position.


  - 0011 Normalize SQL/RPR standard references across code,
         comments, and tests
      Doc/comment/test-header hygiene.  Standardizes citations
      to ISO/IEC 19075-5 (SQL Technical Report Part 5, "Row
      pattern recognition in SQL"); earlier comments mixed bare
      "19075-5", "9075-2", "SQL standard", and "SQL:2016 STR06"
      forms.  Where Chapter 4 (FROM / R010) and Chapter 6
      (WINDOW / R020) describe parallel material, cites the
      Chapter 6 subclause first since this implementation
      targets R020.  Pins "STR06" to its source (Subclause
      7.2.8) in the rpr_nfa / rpr_explain commentary; replaces
      ad-hoc section / arrow glyphs with ASCII.  Adds a short
      normative-reference paragraph to executor/README.rpr so
      future patches inherit the policy.  Comments / docs /
      test headers only.


Still deferred:
  - B7 Recursive CTE XXX: pending community input on the
    ISO/IEC 19075-5 6.17.5 / 4.18.5 interpretation.


Best regards,
Henson

  327. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-05-10T02:54:37Z

    Hi Henson,

> Still deferred:
>   - B7 Recursive CTE XXX: pending community input on the
>     ISO/IEC 19075-5 6.17.5 / 4.18.5 interpretation.

ISO/IEC 19075-5 6.17.5 stats that "Row pattern matching is prohibited
in recursive queries. For example, the following is a syntax error:

CREATE RECURSIVE VIEW Problem... SELECT Kolo, Xoro FROM Ticker... WINDOW W AS... MEASURES..."

ISO/IEC 19075-5 4.18.5 stats that "Row pattern matching is prohibited
in recursive queries. For example, the following is a syntax error:

CREATE RECURSIVE VIEW Problem... SELECT Kolo, Xoro FROM t... MATCH_REZOGNIZE..."

From these it is apparent that CREATE RECURSIVE VIEW cannot be used
with RPR in both R010 and R020.

Question is, what about CTE queries?

I looked into ISO/IEC 9075-2:2016 (I don't have access to 2023) and
found this in "7.17 <query expression>".

in "Syntax Rule"

3) If <with clause> WC is specified, then:

a) Let n be the number of <with list element>s.
:
:
3) If <with clause> WC is specified, then:
:
:
e) If WC immediately contains RECURSIVE, then WC, its <with list>, and
   its <with list element>s are said to be potentially
   recursive. Otherwise, they are said to be non-recursive.

f) A potentially recursive <with list element> shall not contain a
   <row pattern measures> or <row pattern common syntax>.

So I think at least SQL:2016 explicitly prohibits using RPR within
recursive CTE. I would appreciate if anybody confirms this in
SQL:2023.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  328. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-05-12T06:03:28Z

    Hi Henson,

> Attached is the v47 patches for Row pattern recognition (SQL/RPR).

> - Add src/backend/executor/README.rpr (previously was in ExecRPR.c)

README.rpr is extremely useful for those who want to review the RPR
patches.  I found a room to enhance the document.  Attached is a small
patch tries to enhance README.rpr, on top of v47.

- Make "target audience" and "scope" of the README more descriptive.
- Add References (currently the SQL standards only)
- Add explanation of some abbreviations (NFA, AST)
- Add reference sections for absorption. Readers might not be familiar with "absorption"
- Add more fields to WindowAggState
- Add window framing rules with RPR

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  329. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-05-12T07:01:05Z

    Hi Tatsuo,


Four more incremental patches on top of the 2026-05-09 set
(0001-0011).  0012-0013 close the previously deferred B7
recursive-CTE item, on the strength of your ISO/IEC 9075-2:2016
7.17 citation; 0014-0015 apply and round out the README.rpr
enhancement you sent on 2026-05-12.


Patch summary:


> So I think at least SQL:2016 explicitly prohibits using RPR within
> recursive CTE. I would appreciate if anybody confirms this in
> SQL:2023.

  - 0012 Add rpr_integration B7 cases for RPR in recursive
         query
      Test-only prelude to 0013.  Replaces the prior B7 case
      (RPR in a recursive CTE base leg, asserting it works)
      with two cases that the upcoming prohibition needs to
      cover: WITH RECURSIVE with RPR in the base leg, and
      CREATE RECURSIVE VIEW with an RPR window.  Cites
      ISO/IEC 19075-5 6.17.5 (R020) and 4.18.5 (R010), and
      the formal rule in ISO/IEC 9075-2:2016 7.17 Syntax
      Rule 3)e)f) you pointed me to.  Drops the deferred
      XXX comment that left this open to community input.
      Expected output still matches pre-rejection behavior;
      0013 flips both queries to ERROR.


  - 0013 Reject row pattern recognition in recursive queries
      Per ISO/IEC 9075-2:2016 7.17 Syntax Rule 3)e)f), every
      <with list element> in a WITH RECURSIVE clause is
      "potentially recursive" and shall not contain a
      <row pattern common syntax>.  ISO/IEC 19075-5 6.17.5
      (R020) and 4.18.5 (R010) restate the prohibition for
      CREATE RECURSIVE VIEW, which makeRecursiveViewSelect()
      already rewrites to WITH RECURSIVE so the same path
      catches both forms.  The rejection runs in
      transformWithClause() against the raw parse tree,
      before per-CTE analysis, and reports the PATTERN keyword
      position via a new RPCommonSyntax.location field
      captured in gram.y.  Flips both B7 cases from result
      rows to the new error.


> Attached is the v47 patches for Row pattern recognition (SQL/RPR).
>
> > - Add src/backend/executor/README.rpr (previously was in ExecRPR.c)
>
> README.rpr is extremely useful for those who want to review the RPR
> patches.  I found a room to enhance the document.  Attached is a small
> patch tries to enhance README.rpr, on top of v47.
>
> - Make "target audience" and "scope" of the README more descriptive.
> - Add References (currently the SQL standards only)
> - Add explanation of some abbreviations (NFA, AST)
> - Add reference sections for absorption. Readers might not be familiar
> with "absorption"
> - Add more fields to WindowAggState
> - Add window framing rules with RPR
>

  - 0014 Enhance README.rpr per Tatsuo Ishii's review
      Doc-only.  Applies your off-list enhancement patch
      (2026-05-12) on top of v47 with no modification:
        * Make "target audience" and "scope" more
          descriptive, pointing readers to the SQL standard
          (with Oracle / Trino manuals as alternatives)
        * Spell out NFA and AST on first use
        * Cross-reference "IV-5. Absorbability Analysis" and
          "VIII-2. Solution: Context Absorption" from the
          RPR_ELEM_ABSORBABLE_BRANCH flag description
        * List nfaVisitedNWords, defineMatchStartDependent,
          and nfaLastProcessedRow in V-3
        * State the window framing rules that apply with RPR
        * Add a References section (SQL standards)


  - 0015 Round out README.rpr WindowAggState field coverage
      Doc-only follow-up to 0014.  Completes the V-3 field
      list (adds nfaVisitedMinWord / nfaVisitedMaxWord; notes
      that EXPLAIN ANALYZE instrumentation counters are
      intentionally omitted, with a pointer to execnodes.h)
      and mirrors the new fields in the Appendix B diagram,
      which still reflected the pre-review field list.


Previously deferred items:
  - B7 Recursive CTE XXX: addressed in 0012 (tests) + 0013
    (rejection) per the standards citations you supplied.


Best regards,
Henson

  330. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-05-13T03:11:14Z

    Hi Henson,

Thank you for the patches. I looked into 0013.

> diff --git a/src/backend/parser/parse_cte.c b/src/backend/parser/parse_cte.c
> index ccde199319a..3830597bb2b 100644
> --- a/src/backend/parser/parse_cte.c
> +++ b/src/backend/parser/parse_cte.c
> @@ -96,6 +96,14 @@ static void checkWellFormedRecursion(CteState *cstate);
>  static bool checkWellFormedRecursionWalker(Node *node, CteState *cstate);
>  static void checkWellFormedSelectStmt(SelectStmt *stmt, CteState *cstate);
>  
> +/* Recursive-WITH RPR rejection */
> +typedef struct
> +{
> +	ParseLoc	location;		/* location of first RPR window, or -1 */
> +} ContainRPRContext;
> +
> +static bool contain_rpr_walker(Node *node, void *context);
> +
>  
>  /*
>   * transformWithClause -
> @@ -157,13 +165,31 @@ transformWithClause(ParseState *pstate, WithClause *withClause)
>  	if (withClause->recursive)
>  	{
>  		/*
> -		 * For WITH RECURSIVE, we rearrange the list elements if needed to
> -		 * eliminate forward references.  First, build a work array and set up
> -		 * the data structure needed by the tree walkers.

I think removing the exiting comment ("For WITH RECURSIVE, we rearrange
the list elements...") is not appropriate as this explains subsequent
process, which is not changed after the patch.

> +		 * Per ISO/IEC 9075-2:2016 7.17 Syntax Rule 3)e)f), every <with list
> +		 * element> in a WITH RECURSIVE clause is "potentially recursive" and
> +		 * shall not contain a <row pattern common syntax>.  (PostgreSQL does
> +		 * not implement <row pattern measures>, so only the common syntax
> +		 * needs to be checked.)  ISO/IEC 19075-5 6.17.5 (R020) and 4.18.5
> +		 * (R010) restate the prohibition for CREATE RECURSIVE VIEW, which is
> +		 * rewritten to WITH RECURSIVE by makeRecursiveViewSelect() and so
> +		 * flows through here as well.
>  		 */
>  		CteState	cstate;
>  		int			i;
>  
> +		foreach(lc, withClause->ctes)
> +		{
> +			CommonTableExpr *cte = (CommonTableExpr *) lfirst(lc);
> +			ContainRPRContext ctx;
> +
> +			ctx.location = -1;
> +			if (contain_rpr_walker(cte->ctequery, &ctx))
> +				ereport(ERROR,
> +						errcode(ERRCODE_FEATURE_NOT_SUPPORTED),

ERRCODE_FEATURE_NOT_SUPPORTED should be ERRCODE_SYNTAX_ERROR instead?
IMO ERRCODE_FEATURE_NOT_SUPPORTED is used when the feature is defined
by the standard but PostgreSQL just has not implemented yet. In this
case the standard disllow RPR in recursive CTE.

> +						errmsg("cannot use row pattern recognition in a recursive query"),
> +						parser_errposition(pstate, ctx.location));
> +		}
> +
>  		cstate.pstate = pstate;
>  		cstate.numitems = list_length(withClause->ctes);
>  		cstate.items = (CteItem *) palloc0(cstate.numitems * sizeof(CteItem));
> @@ -1268,3 +1294,29 @@ checkWellFormedSelectStmt(SelectStmt *stmt, CteState *cstate)
>  		}
>  	}
>  }

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  331. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-05-13T04:15:20Z

    Hi Tatsuo,

Thanks for the review.  Both points addressed in the attached
0013 v2; 0012/0014/0015 are unchanged from the 2026-05-12 set
and re-attached for convenience.

I think removing the exiting comment ("For WITH RECURSIVE, we rearrange
> the list elements...") is not appropriate as this explains subsequent
> process, which is not changed after the patch.
>

Agreed -- the original comment still describes the work-array
and tree-walker setup that follows, which the patch does not
touch.  Restored it in place and moved the new ISO citation
comment to sit directly above the RPR rejection foreach so
each block is documented next to the code it explains.


> ERRCODE_FEATURE_NOT_SUPPORTED should be ERRCODE_SYNTAX_ERROR instead?
> IMO ERRCODE_FEATURE_NOT_SUPPORTED is used when the feature is defined
> by the standard but PostgreSQL just has not implemented yet. In this
> case the standard disllow RPR in recursive CTE.


Right, the standard explicitly prohibits this combination
rather than leaving it unimplemented.  Switched to
ERRCODE_SYNTAX_ERROR.  The .out files reference only the
ERROR message so the regress expected output is unchanged.

Thanks,
Henson

  332. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-05-17T10:00:23Z

    Hi Jian,

>> > Attached is the v47 patches for Row pattern recognition (SQL/RPR).
>>
> 
> Hi
> The patchset is too big, I only downloaded v47-0001...

I understand your complain. Yes, the patch set is big. But that's
because the feature (SQL/RPR) is big. There's no way to avoid it.  I
tried to make the patch set as small as possible, keeping the
functionality to be useful in the real world at the same time.

Fortunately Henson wrote a very usefully document:
src/backend/executor/README.rpr. It draws an outline of overall
architecture of the patch set. I strongly recommend to read it before
jumping into the code.

Another useful documents are the SQL standards. I recommend to read
ISO/IEC 19075-5. It's a 73-page document (as of 2021 edition), which
is relatively small volume comparing with other standard documents.

Currently the patch set implements PATTERN and DEFINE clauses along
with NFA regular expression engine. It lacks MEASURES clause and range
var qualified row pattern variables. If we implement these features,
it will make the patch set size even bigger. So we think that it would
be better to implement them after the initial commit of RPR.  (Of
course this does not prevent the discussion of designing MEASURES
etc. for the future. We welcome it)

If you still want to add some features, there are moderately smaller
ones. For example, SEEK clause, EMPTY MATCH and subquery in the DEFINE
clause.

Thank you for interesting in RPR!

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

> Just a quick skim of the code on my end.
> I noticed two code style issues, currently we don't have a code style
> reference in the manual.
> 
> 1. We need to add trailing commas to enum definitions. See
> https://git.postgresql.org/cgit/postgresql.git/commit/?id=611806cd726fc92989ac918eac48fd8d684869c7
> 
> 2.
> + | '{' ',' Iconst '}' Op
> + {
> + if (strcmp($5, "?") != 0)
> + ereport(ERROR,
> + (errcode(ERRCODE_SYNTAX_ERROR),
> + errmsg("invalid token after range quantifier"),
> + errhint("Only \"?\" is allowed after {n,} or {,m} to make it reluctant."),
> + parser_errposition(@5)));
> + if ($3 <= 0 || $3 >= INT_MAX)
> + ereport(ERROR,
> + errcode(ERRCODE_SYNTAX_ERROR),
> + errmsg("quantifier bound must be between 1 and %d", INT_MAX - 1),
> + parser_errposition(@3));
> + $$ = (Node *) makeRPRQuantifier(0, $3, @5, @1, yyscanner);
> 
> + (errcode(ERRCODE_SYNTAX_ERROR),
> The leading parenthesis is optional and can be removed, fewer
> parentheses are always better.
> See related discussion:
> https://postgr.es/m/202510100916.s2e6n3xiwvyc@alvherre.pgsql
> 
> Since v47-0001 introduces many ereport(ERROR) messages, it makes sense
> to move the related regression tests from the other patch into
> v47-0001, IMHO.
> 
> 
> 
> --
> jian
> https://www.enterprisedb.com/

  333. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-05-24T08:11:39Z

    Hi Henson,

> Attached is the v47 patches for Row pattern recognition (SQL/RPR).

While looking into v47, I noticed that
raw_expression_tree_walker_impl() lacks tracking RPCommonSyntax and
its children nodes. Probably this does nothing wrong with RPR
functionalities but just for completeness I created a patch on top of
v47.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  334. Re: Row pattern recognition

    jian he <jian.universality@gmail.com> — 2026-05-26T03:20:04Z

    Hi.

Disclaimer: I had some private off-list discussions with Henson Choi
regarding execRPR.c, and the NFA.
The following are more formal comments regarding execRPR.c and the NFA
algorithm, based on v47-0001 to v47-0009, I haven't applied the
incremental diff yet.
(Since the incremental diffs are scattered across several threads, a
unified v48 would be better).
(I'm still wrapping my head around the NFA in execRPR.c, so take the
comments below with a grain of salt.)


PATTERN (A B C D)
We can short-circuit and exit early if any of the evaluations (A, B,
or C) fail in nfa_match.
This is necessary since the chance of a pattern element evaluation
returning false is not rare, I think.


For src/backend/executor/README.rpr:
We should explicitly explain 'absorbable' and 'absorption' somewhere in
README.rpr, as the text currently just assumes the reader knows what they mean.
Using some example illustrate "absorption" meaning, put it on
README.rpr would be great.
We can also mention that 'DFS' refers to Depth-First Search".

``````
  (4) Call nfa_advance(initialAdvance=true)
``````
In V47, the variable `initialAdvance` does not exist.

In nfa_advance_var, after the first Assert, we can add:
Assert(elem->next <  pattern->numElements);

ExecRPRFinalizeAllContexts seems unnecessary; I commented it out,
rerun the regress tests
(TESTS='test_setup rpr_base rpr_nfa rpr_explain rpr_integration rpr'
meson test -C $BUILD3 --num-processes 20 --suite regress --verbose)
Only two SQL tests in rpr_explain.sql failed.

SELECT first_value(id) OVER w AS match_start FROM stock_ticks
WINDOW w AS ( ORDER BY id ROWS BETWEEN CURRENT ROW AND UNBOUNDED
FOLLOWING AFTER MATCH SKIP PAST LAST ROW PATTERN ((A B) {2}) DEFINE A
AS price < 100, B AS price < 100);

The query above invokes the following code. Since the PATTERN above is
not greedy, is the comment below incorrect?
``````
    else
    {
        /* Greedy: enter first, skip second */
        state->elemIdx = elem->next;
        nfa_route_to_elem(winstate, ctx, state,
                          &elements[state->elemIdx], currentPos);
        if (skipState != NULL)
        {
            nfa_route_to_elem(winstate, ctx, skipState,
                              &elements[elem->jump], currentPos);
        }
    }
``````

nfa_advance_var
```
    else if (canExit)
    {
        state->counts[depth] = 0;
        state->elemIdx = elem->next;
        nextElem = &elements[state->elemIdx];
        /*
         * Update isAbsorbable for target element (monotonic: AND preserves
         * false)
         */
        state->isAbsorbable = state->isAbsorbable &&
            RPRElemIsAbsorbableBranch(nextElem);
        /* See comment above: increment outer END count for quantified VARs */
        if (RPRElemIsEnd(nextElem))
        {
            if (state->counts[nextElem->depth] < RPR_COUNT_MAX)
                state->counts[nextElem->depth]++;
        }
        nfa_route_to_elem(winstate, ctx, state, nextElem, currentPos);
    }
```
The above ELSE IF overrides all RPRNFAState field values except
RPRNFAState->next.
Should we set RPRNFAState->next to NULL?
(If I add ``state->next = NULL;`` in the above ELSE IF branch, all the
regress tests still pass)
Some comments about the situation of (RPRNFAState) state->next would be great.
This also happens in `` if (canLoop && canExit)``, ``if (reluctant)`` branch.

For function nfa_advance_var, I don't understand the meaning of the
variable "count", after the first Assert I have added below:

    if (count > 2 && !state->isAbsorbable && RPRElemIsReluctant(elem))
        elog(INFO, "count is %d and elem is reluctant and isAbsorbable
is false XXXXX", count);
    if (count > 2 && state->isAbsorbable && RPRElemIsReluctant(elem))
        elog(INFO, "count is %d and elem is reluctant and isAbsorbable
is true XXXXX", count);

Rerunning the regress tests shows that count >= 3 occurs very infrequently.
I'm don't understand isAbsorbable, so I am not sure if the second
``elog(INFO`` is actually reachable or not.
Can we add more complex queries (more count >= 3) to check if the
"count" variable is working correctly?


In function nfa_add_state_unique:
    /* Mark VAR in visited before duplicate check to prevent DFS loops */
    winstate->nfaVisitedElems[WORDNUM(state->elemIdx)] |=
        ((bitmapword) 1 << BITNUM(state->elemIdx));

I honestly don't understand the purpose of the code block above. But it doesn't
seem to influence the subsequent FOR LOOP;
It only modifies its own variables without any other side effects within
nfa_add_state_unique. Could we add some comments explaining which external
functions rely on this code and why it belongs in nfa_add_state_unique?


nfa_states_equal
compareDepth = elem->depth + 1; /* depth 0 needs 1 count, etc. */
The comment above isn't helpful, IMHO, and I don't understand it.
We should focus on why compareDepth should be ```elem->depth + 1```.


function nfa_add_state_unique return value is not being used?
Do we need to do something with the return value, or is this expected?
(I don't have an opinion on it, I guess it would be better to raise this issue)


In nfa_advance_alt, during the main WHILE loop, I think altElem->depth
must be larger than elem->depth.
Therefore we can do
``````
        if (altElem->depth == elem->depth)
            elog(ERROR, "nfa_advance_alt altElem->depth should not be
the same as elem->depth reached");
        if (altElem->depth < elem->depth)
            break;
``````



--
jian
https://www.enterprisedb.com/

  335. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-05-26T22:31:02Z

    Hi Tatsuo, jian,

While looking into v47, I noticed that
> raw_expression_tree_walker_impl() lacks tracking RPCommonSyntax and
> its children nodes. Probably this does nothing wrong with RPR
> functionalities but just for completeness I created a patch on top of
> v47.


Thanks for catching this. I applied the patch and the RPR regress passes
cleanly. I also re-ran it with debug_raw_expression_coverage_test turned
on (on an assert-enabled build), and the full regress is green as well.

One observation while testing: the GUC catches missing case handlers
once the walker actually reaches a node, but it cannot flag a missing
WALK on its own -- if no caller drives the walker into a subtree, the
omission stays silent. So your inspection was the part that found the
gap; the GUC just confirms the patch closes it. With the patch in, RPR
raw subtrees are on the safety net for any future node-type additions.

I'll include the patch in v48 as nocfbot-0015. My suggestion would
be to defer the fold until the jian-response patches (numbered from
0016 onward, which I'll be sending shortly) have also gone through a
review round, so the whole bundle can land together in one pass.
Patch attached below for convenience.


jian -- thanks for the thorough review. It covers a lot of ground,
and I'm still working through it. Current expectation is that most
items will be accepted; a few smaller ones may end up with a
different conclusion than your suggestion, and those are still under
analysis on my side.

The plan is to turn the responses into a patch series and send them
out for another round of review. More to follow once the batch is in
shape.


Thanks,
Henson

  336. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-05-27T02:49:06Z

    Hi Henson,

> Thanks for catching this. I applied the patch and the RPR regress passes
> cleanly. I also re-ran it with debug_raw_expression_coverage_test turned
> on (on an assert-enabled build), and the full regress is green as well.
> 
> One observation while testing: the GUC catches missing case handlers
> once the walker actually reaches a node, but it cannot flag a missing
> WALK on its own -- if no caller drives the walker into a subtree, the
> omission stays silent. So your inspection was the part that found the
> gap; the GUC just confirms the patch closes it. With the patch in, RPR
> raw subtrees are on the safety net for any future node-type additions.

Thanks for checking.

> I'll include the patch in v48 as nocfbot-0015. My suggestion would
> be to defer the fold until the jian-response patches (numbered from
> 0016 onward, which I'll be sending shortly) have also gone through a
> review round, so the whole bundle can land together in one pass.
> Patch attached below for convenience.

Sure. I will wait for your ready-to-go signal before creating the v48
pach set.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  337. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-05-27T05:49:44Z

    Hi jian, Tatsuo,

Thanks for the thorough first read of execRPR.c and the NFA. The
depth of understanding you reached on a fairly intricate body of
code in a short window is impressive -- and several of the open
questions in your review pointed straight at documentation and
comment debt on our side that needed exposing. First-reviewer
feedback of that kind is especially valuable, because the gaps it
surfaces are exactly the ones the next reader would otherwise have
to re-discover; closing them now puts everyone reading the code
afterwards on firmer ground. Would be glad to keep working together
on this area.


> Disclaimer: I had some private off-list discussions with Henson Choi
> regarding execRPR.c, and the NFA.
> The following are more formal comments regarding execRPR.c and the NFA
> algorithm, based on v47-0001 to v47-0009, I haven't applied the
> incremental diff yet.
> (Since the incremental diffs are scattered across several threads, a
> unified v48 would be better).
> (I'm still wrapping my head around the NFA in execRPR.c, so take the
> comments below with a grain of salt.)

Agreed on the unified v48 -- both the scattered incrementals and
the responses to your review will be part of that series. The
responses below will land alongside their corresponding change
(README, comment, test, or commit) in that series rather than as
a free-standing reply that defers the actual fixes.


> PATTERN (A B C D)
> We can short-circuit and exit early if any of the evaluations (A, B,
> or C) fail in nfa_match.
> This is necessary since the chance of a pattern element evaluation
> returning false is not rare, I think.

Agreed on the optimization intent. A slightly different shape worth
considering: rather than eager short-circuit in nfa_match, defer
DEFINE predicate evaluation to first use -- varMatched[i] computed
on demand and cached per row, so pruned paths never pay for
predicates they didn't reach. The code change is small; the part
needing care is whether "not evaluating" is safe for every predicate
(navigation state, slot setup, side effects). The current code or
the standard may already enforce constraints that make lazy
evaluation naturally safe, but I'd rather not act on that assumption
without concrete grounding in what the code actually enforces or in
test coverage that pins it down.

Would you be interested in driving the discussion in this thread?
You've been deep in execRPR.c recently, so the trade-offs sit
closest to you, and I think Tatsuo will land a good conclusion once
they're on the table. Happy to support the discussion in whatever
way is useful as it converges.


> For src/backend/executor/README.rpr:
> We should explicitly explain 'absorbable' and 'absorption' somewhere in
> README.rpr, as the text currently just assumes the reader knows what they
mean.
> Using some example illustrate "absorption" meaning, put it on
> README.rpr would be great.
> We can also mention that 'DFS' refers to Depth-First Search".

Acknowledged, and the request surfaced an underlying problem in the
README's terminology. "Absorption" is currently used for two
distinct things: an AST-level rewrite in Phase 1 that pulls
identical sequences around a group inside it, and the runtime
context-equivalence collapse that drives the O(n^2) -> O(n)
optimization. Sharing the word leaves a reader encountering
"absorbable" early on without an anchor.

Rather than disambiguate by qualifier ("prefix/suffix absorption"
vs "context absorption"), I'd lean toward renaming the AST-level
case so "absorption" stays reserved for the runtime concept. The
README then only needs to explain absorption in one place, in
detail, without the disambig preamble.

For the rename, "prefix/suffix merging" feels like the natural fit
-- the other AST-level optimizations in the same Phase 1 are already
named "consecutive variable / group / ALT merging", so it slots in
cleanly. "Prefix/suffix factoring" is another candidate if a more
descriptive verb is preferred.

Tatsuo, curious what you think of this direction and naming. Happy
to take any name you prefer for the AST-level operation, or to keep
the original "absorption" wording with stronger forward-references
if you'd rather not rename.

For the absorption explanation itself in README.rpr, the diagnosis
I'd offer is that Chapter VIII already carries the necessary content
-- the issue is narrative order. VIII-1 leads with the O(n^2) problem
framing, so a reader meets the cost shape before meeting the
intuition for why absorption is possible, and has to carry the
problem until VIII-2's monotonicity argument finally lands. Beyond
that, VIII-2 stays abstract; there is no row-by-row trace showing
two states being judged equivalent.

Two small additions seem to close most of the gap:

  (A) A 4-5 line intuition summary at the top of Chapter VIII,
      before VIII-1, naming what absorption is (collapsing contexts
      that have converged on identical future behavior) and the
      monotonicity principle that makes it safe. This gives the
      reader an anchor before the problem framing.

  (B) A short worked example at the end of VIII-2: a PATTERN (A+)
      trace over a few rows showing each new context being absorbed
      by Context_1 once its (elemIdx, depth-0 count) is dominated.
      Concrete state/count comparisons make the abstract solution
      land.

Curious if this read of the gap matches what tripped you up, and
whether (A) + (B) feel sufficient. Happy to draft both as part of
the v48 README changes.

For DFS, will expand it to "Depth-First Search (DFS)" at the first
occurrence.


> ``````
>   (4) Call nfa_advance(initialAdvance=true)
> ``````
> In V47, the variable `initialAdvance` does not exist.

Leftover from an earlier patch version -- the boolean parameter was
refactored away and the README notation wasn't updated. I'll bring
it in line with the current signature.


> In nfa_advance_var, after the first Assert, we can add:
> Assert(elem->next <  pattern->numElements);

Agreed. Will add it right after Assert(canLoop || canExit) in
nfa_advance_var, with a >= 0 lower bound tacked on while there
(RPRElemIdx is signed int16, INVALID = -1):

  Assert(elem->next >= 0 && elem->next < pattern->numElements);


> ExecRPRFinalizeAllContexts seems unnecessary; I commented it out,
> rerun the regress tests
> (TESTS='test_setup rpr_base rpr_nfa rpr_explain rpr_integration rpr'
> meson test -C $BUILD3 --num-processes 20 --suite regress --verbose)
> Only two SQL tests in rpr_explain.sql failed.

Reproduced this. You're right on correctness and memory: data rows
are identical with the call removed, and release_partition's
MemoryContextReset reclaims memory anyway.

Finalize isn't really about handling matches. By the time the
partition ends, all genuine FIN reaches have already been recorded
in-flight. Its job is to kill any VAR states still pursuing when
rows run out, so cleanup sees a uniform ctx->states == NULL across
every context. Three shapes survive there:

  - Pure pursuit (no matchedState, e.g., A+ B mid-pattern).
  - Empty-match candidate + pursuit (matchedState set with
    matchEndRow < matchStartRow -- e.g., greedy A* with no
    successful matches yet, while VAR is still chasing a longer
    non-empty one).
  - Real match + pursuit (matchedState set with matchEndRow >=
    matchStartRow -- e.g., greedy A* with some matches recorded
    and still looping for a longer one).

The first two get reclassified as failures by cleanup; without
Finalize they linger without contributing to stats. The third is
stat-neutral -- cleanup skips it either way -- but goes through
the same uniform path so partition-end classification stays
centralized.

The classification surfaces today only via rpr_explain stats, but
becomes user-visible once we extend the R020 surface or move into
R010 -- MEASURES and eventual R010 hooks count matches based on
this classification. Worth locking in now, and an explicit
partition-end stage is structurally cleaner than scattering the
logic.

Plan: keep the call and reframe it as the partition-end
classification policy holder. Strategically, that gives the future
partition-end hooks a single anchor to extend, instead of growing
scattered end-of-partition paths.


> SELECT first_value(id) OVER w AS match_start FROM stock_ticks
> WINDOW w AS ( ORDER BY id ROWS BETWEEN CURRENT ROW AND UNBOUNDED
> FOLLOWING AFTER MATCH SKIP PAST LAST ROW PATTERN ((A B) {2}) DEFINE A
> AS price < 100, B AS price < 100);
>
> The query above invokes the following code. Since the PATTERN above is
> not greedy, is the comment below incorrect?
> ``````
>     else
>     {
>         /* Greedy: enter first, skip second */
>         ...
>     }
> ``````

The comment is misleading; the code is correct. Two pieces:

First, greedy vs reluctant: '?' plays two distinct roles in our
grammar. As a quantifier on its own (A?, (A B)?) it means "optional"
-- equivalent to {0,1}. As a suffix on another quantifier (A+?,
(A B)*?, {n}?, etc.) it makes that quantifier reluctant. So {n}
without a trailing '?' is greedy; the {n}? form is reluctant.
PATTERN ((A B){2}) is in fact greedy.

Second, why the branch is entered: nfa_advance_begin's else arm
handles two cases at once:

  (a) Greedy with optional group: skipState != NULL, not reluctant.
      "Enter the group; also create the skip path."
  (b) Non-nullable group (min > 0, regardless of greedy/reluctant):
      skipState stays NULL, the outer guard
      "if (skipState != NULL && RPRElemIsReluctant(elem))" falls
      through, and the inner "if (skipState != NULL)" prevents the
      skip-path action from running.

(A B){2} has min = max = 2, so it lands in (b) -- the action that
actually runs is "enter the group", no skip path. The current label
only describes (a), which is why it reads wrong for your test query.
Plan is to rewrite the comment along the lines of
"Greedy-or-non-nullable: route to the first child; for optional
groups (skipState != NULL), additionally create the skip path."


> nfa_advance_var
> ```
>     else if (canExit)
>     {
>         ...
>     }
> ```
> The above ELSE IF overrides all RPRNFAState field values except
> RPRNFAState->next.
> Should we set RPRNFAState->next to NULL?
> (If I add ``state->next = NULL;`` in the above ELSE IF branch, all the
> regress tests still pass)

Good catch on the asymmetry. Tracing it through, state->next is
actually already NULL at every branch you flagged: nfa_advance
resets it just before crossing into nfa_advance_state, and the
intermediate branches don't disturb it. Your experiment passing
with an added "state->next = NULL" is consistent with that -- the
assignment is redundant rather than load-bearing.

The contract that keeps state->next sane lives at two concentrated
points (nfa_advance entry, nfa_add_state_unique linking), and the
branches in between are pass-through. Sprinkling the same reset at
every branch would be defensive noise rather than a real safety
net, so I'd leave the branches alone.

Happy to add a short comment near nfa_advance's reset marking it
as the boundary contract, so the next reader doesn't trip on the
same question.


> For function nfa_advance_var, I don't understand the meaning of the
> variable "count", after the first Assert I have added below:
> ...
> Rerunning the regress tests shows that count >= 3 occurs very
infrequently.
> ...
> Can we add more complex queries (more count >= 3) to check if the
> "count" variable is working correctly?

The "count" semantics will read more cleanly once the absorption
README work above lands (counts[d] = iteration count at nesting
depth d). For coverage I'll add a nested reluctant quantifier to
rpr_nfa (e.g., PATTERN ((A B){3,5}? C)) to drive count through the
3..5 band repeatedly. (rpr_nfa is the suite that already targets
Quantifier Runtime Behavior and Absorption Optimization.)


> In function nfa_add_state_unique:
>     /* Mark VAR in visited before duplicate check to prevent DFS loops */
>     ...
> I honestly don't understand the purpose of the code block above. But it
doesn't
> seem to influence the subsequent FOR LOOP;
> ...
> Could we add some comments explaining which external functions rely on
> this code and why it belongs in nfa_add_state_unique?

The code is correct, but the contract is split across two functions
and currently only one side points to the other. The visited marking
scheme is asymmetric on purpose:

  - Non-VAR elements (END/ALT/BEGIN/FIN) are marked on entry to
    nfa_advance_state because epsilon cycles must be prevented
    immediately.
  - VAR elements are marked later, in nfa_add_state_unique, only
    when added to the state list. That delay is intentional: it
    keeps legitimate quantifier loop-back to the same VAR across
    iterations possible.

The paired cycle check sits in nfa_advance_state, and the
asymmetric-marking rationale is documented there. What's missing is
the back-reference from nfa_add_state_unique. I'll add a single line
at the marking site pointing back to nfa_advance_state.


> nfa_states_equal
> compareDepth = elem->depth + 1; /* depth 0 needs 1 count, etc. */
> The comment above isn't helpful, IMHO, and I don't understand it.
> We should focus on why compareDepth should be ```elem->depth + 1```.

Agree the trailing comment is too terse. Two pieces are missing:

  (a) The +1 arithmetic: to compare counts up to depth N, we need
      slots counts[0..N], which is N+1 entries.
  (b) Why deeper slots are excluded: counts[d > elem->depth] are
      scratch state from deeper groups and get reset on re-entry,
      so they must not participate in equivalence judgment.

Two states sharing elemIdx are equivalent iff all
enclosing-or-current depth counts match. I'll replace the trailing
comment with a small block covering both pieces.


> function nfa_add_state_unique return value is not being used?
> Do we need to do something with the return value, or is this expected?
> (I don't have an opinion on it, I guess it would be better to raise this
issue)

Leftover from an earlier design -- the duplicate case is fully
handled inside the function (the state is freed and nfaStatesMerged
is incremented), so callers have nothing to branch on, and indeed
none of them do. Will change the signature from bool to void and
drop the return statements.


> In nfa_advance_alt, during the main WHILE loop, I think altElem->depth
> must be larger than elem->depth.
> Therefore we can do
> ``````
>         if (altElem->depth == elem->depth)
>             elog(ERROR, "nfa_advance_alt altElem->depth should not be
> the same as elem->depth reached");
>         if (altElem->depth < elem->depth)
>             break;
> ``````

I had to push back on this one. Tracing the depth bookkeeping:

  - For an ALT at depth D, branches sit at depth D+1, and each
    branch's first element has .jump pointing to the next branch's
    first (set in fillRPRPatternAlt). So the walk normally
    terminates when the last branch's .jump = INVALID -- the depth
    check doesn't fire at all.
  - But when the last branch is a quantified group, its first
    element is a BEGIN whose .jump = past-END (set by
    fillRPRPatternGroup and not overridden for the last branch).
    The walk then steps to a post-ALT element, and the depth check
    is what stops it from creating a stray state out there.

That post-ALT element has depth <= D:

  * D-1 if the ALT is inside an enclosing group with a non-trivial
    quantifier, e.g., PATTERN ((A | (B C)+){2}) -- post-ALT lands
    on the outer END at depth 0, ALT at depth 1. (A {1,1} outer
    wrap gets removed by single-child unwrap, so it has to be a
    real quantifier.)
  * D if the ALT has a sibling at the same level, e.g.,
    PATTERN (A | (B C)+) at top level -- post-ALT is FIN at depth 0,
    matching the ALT's depth 0.

So "altElem->depth == elem->depth" is a legitimate end-of-walk
signal for the quantified-group-last-branch case, not an invariant
violation. Treating it as an error would misfire on patterns like
A | (B C)+. The current "if (altElem->depth <= elem->depth) break;"
in nfa_advance_alt is intentionally <= and not <, and the looser
comparison is correct. Happy to add a brief comment there noting
the trigger condition, if it would help future readers.


Summary of decisions, in the order above:

  Short-circuit optimization       Separate series -- invite you to drive
  Absorption README narrative      Accept -- Chapter VIII summary + example
  AST-level "absorption" rename    Pending Tatsuo's call -- prefix/suffix
merging?
  DFS expansion                    Accept
  initialAdvance README mismatch   Accept -- align with current signature
  Defensive Assert in advance_var  Accept -- also add lower bound
  Finalize unnecessary?            Keep -- partition-end policy holder
  Greedy comment label             Accept -- rewrite to cover both cases
  state->next reset                Decline -- boundary contract covers it
  count >= 3 test coverage         Accept -- add to rpr_nfa
  visited marking purpose          Accept -- add back-reference comment
  compareDepth comment             Accept -- rewrite with intent
  Unused bool return               Accept -- change to void
  ALT depth invariant Assert       Decline -- end-of-walk signal, not
invariant

That's the full pass. The actual patches (nocfbot-0016 onward) will
follow shortly as a separate submission, for another review round.


Thanks,
Henson

  338. Re: Row pattern recognition

    jian he <jian.universality@gmail.com> — 2026-05-28T13:23:59Z

    Hi.

In src/include/nodes/execnodes.h, we're adding quite a few fields to
WindowAggState that are only used for RPR queries.
Should we consolidate these fields behind a single pointer (named
RPRContext) to keep the WindowAggState size smaller for non-RPR
queries?

In function get_reduced_frame_status, I made the following changes:
``````
bool conditionA;
bool conditionB;
bool conditionC;
bool conditionD;
if (!winstate->rpr_match_valid)
    return RF_NOT_DETERMINED;
conditionA = (pos == start && winstate->rpr_match_matched && length == 0);
conditionB = (pos < start || pos >= start + length);
conditionC = !winstate->rpr_match_matched;
conditionD = (pos == start);
if ((conditionA && conditionB) || (conditionA && conditionC) ||
    (conditionA && conditionD))
{
    if (conditionA)
        elog(INFO, "conditionA is true");
    if (conditionB)
        elog(INFO, "conditionB is true");
    if (conditionC)
        elog(INFO, "conditionC is true");
    if (conditionD)
        elog(INFO, "conditionD is true");
}
if ((conditionB && conditionC) || (conditionB && conditionD) ||
(conditionC && conditionD))
{
    elog(INFO, "more than 2 branch is true, should not be reached");
    if (!(conditionC && conditionD))
        elog(INFO, "not (conditionCD is true)");
    if ((conditionB && conditionD))
        elog(INFO, "conditionB and conditionD is true)");
}
``````
I re-ran the regression tests and noticed it's entirely possible for multiple
conditions to be true at once. Because of this, all the IF blocks in
get_reduced_frame_status (after the initial one) are order-dependent. If I move
one IF statement above another, get_reduced_frame_status may return a
different result.
Is this the expected behavior? It seems like a potential logic flaw.

------------------------------------------------------------------------------------
VIII-3. Absorption Conditions

Planner-time prerequisites (all must hold for absorption to be enabled):

  (a) SKIP PAST LAST ROW.  SKIP TO NEXT ROW creates overlapping
      contexts that cannot be safely absorbed.
  (b) Unbounded frame (ROWS BETWEEN CURRENT ROW AND UNBOUNDED
      FOLLOWING).  Limited frames apply differently to each context,
      breaking the monotonicity principle.
  (c) No match_start-dependent navigation in DEFINE.

      Mechanism: each context has a different matchStartRow, so FIRST
      resolves to a different row for each context at the same
      currentpos.  An earlier context's DEFINE result no longer
      subsumes a later one's, making count-dominance comparison
      invalid.  Rather than comparing matchStartRow at runtime
      (which would complicate the absorb path), any match_start
      dependency disables absorption entirely.

      Navigation content              match_start dep.  absorption
      ------------------------------------------------------------
      No navigation                   none              safe
      PREV/NEXT only                  none              safe
      LAST (no offset)                none              safe
      LAST (with offset)              boundary check    unsafe
      FIRST (any)                     direct            unsafe
      Compound (inner FIRST)          direct            unsafe
      Compound (inner LAST, no off.)  none              safe
      Compound (inner LAST, w/off.)   boundary chk      unsafe
------------------------------------------------------------------------------------
"boundary chk" should be "boundary check".
column "match_start dep.", I don't understand the meaning of "direct"
and "boundary check".
"count-dominance" seems like an invented word, but I could not find
the explanation.
"match_start-dependent" occurred many times, it would also deserve an
explanation.
We can also rename it as match_start_dependent.

----------------------------------------------------
Attached is a minor refactoring of ExecRPRProcessRow, no need for
boolean hasLimitedFrame.
``if (hasLimitedFrame && winstate->endOffsetValue != 0)`` seems wrong
to me, endOffsetValue is a Datum,
and you directly compare it with 0.
see also calculate_frame_offsets.
----------------------------------------------------
ExecRPRProcessRow,
nfa_evaluate_row
```
  if (!window_gettupleslot(winobj, pos, slot))
        return false;            /* No row exists */
```
indicates that it's not possible for (currentPos > ctxFrameEnd).
therefore
`````
    if (currentPos >= ctxFrameEnd)
    {
        /* Frame boundary exceeded: force mismatch */
        nfa_match(winstate, ctx, NULL);
        continue;
    }
`````
should change to ```if (currentPos == ctxFrameEnd)```.
With that, the comment wording "exceeded" seems wrong too.

The comment 'Frame boundary exceeded' also needs updating since
it's no longer exceeding the boundary.
----------------------------------------------------
Since I couldn't generate a coverage report, I am using elog(INFO) to test
reachability. Rerun the regress tests, you will find out that the ELSE IF branch
below is not reached by current regress tests.
``````
    else if (targetCtx->states == NULL)
    {
        /* Context already completed - skip to result registration */
        goto register_result;
        elog(INFO, "XXXX reached");
    }
``````



--
jian
https://www.enterprisedb.com/

  339. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-05-29T01:28:29Z

    Hi Jian,

Thanks for the careful read of execRPR.c and the NFA -- several of these
landed on real gaps in our comments/docs. Going through your points
inline below, with a summary of decisions at the end.


> In src/include/nodes/execnodes.h, we're adding quite a few fields to
> WindowAggState that are only used for RPR queries.
> Should we consolidate these fields behind a single pointer (named
> RPRContext) to keep the WindowAggState size smaller for non-RPR
> queries?

The size win is real -- it's roughly 450-500 bytes per WindowAggState
that every non-RPR window query carries today. But it takes a wide
code change to get there: it reshapes every RPR access path, a
sizable (if mostly mechanical) diff across nodeWindowAgg.c,
execRPR.c, and the explain side.

Tatsuo, as co-author -- do you want this in v48? If you do, I'll
prepare the RPRContext consolidation as an incremental patch for you to
fold into v48. It isn't blocking either way.


> In function get_reduced_frame_status, I made the following changes:
> [conditionA..D probe elided]
> I re-ran the regression tests and noticed it's entirely possible for
> multiple conditions to be true at once. Because of this, all the IF
> blocks in get_reduced_frame_status (after the initial one) are
> order-dependent. If I move one IF statement above another,
> get_reduced_frame_status may return a different result.
> Is this the expected behavior? It seems like a potential logic flaw.

You're right that the conditions aren't mutually exclusive and that
reordering the branches changes the result -- but that's the
cascade-with-early-return idiom, standard in C and used throughout the
backend, not a logic flaw. Each branch is the minimal test given the
negations the earlier returns have already established.

heapam_visibility.c is the canonical example: the HeapTupleSatisfies*
family cascades through ordered early returns, with the later XMAX
branches correct only because the XMIN-committed invariant above holds.

The order is correct, not just idiomatic: update_reduced_frame() only
ever records (matched, length) as (false, 1), (true, 0), or (true, >=1),
and A->B->C->D is the order that classifies exactly those three cases --
reorder it and one of them gets misclassified.

I'll add short "by here, ..." comments between the branches to make the
invariants visible, but keep the structure. If you feel strongly it
should be restructured, I'd defer to Tatsuo on that.


> "boundary chk" should be "boundary check".
> column "match_start dep.", I don't understand the meaning of "direct"
> and "boundary check".
> "count-dominance" seems like an invented word, but I could not find
> the explanation.
> "match_start-dependent" occurred many times, it would also deserve an
> explanation. We can also rename it as match_start_dependent.

All fair. v48 fixes the typo and adds a terminology block to README.rpr
defining count-dominance (the VIII-3 cover condition, named there),
match_start_dependent (renamed to the underscore spelling, to match the
defineMatchStartDependent identifier), and the "direct" vs "boundary
check" navigation distinction.


> Attached is a minor refactoring of ExecRPRProcessRow, no need for
> boolean hasLimitedFrame.
> ``if (hasLimitedFrame && winstate->endOffsetValue != 0)`` seems wrong
> to me, endOffsetValue is a Datum, and you directly compare it with 0.
> see also calculate_frame_offsets.

Good eye on the Datum comparison, and I agree your version reads more
clearly -- gating on the frame-option flag and going through
DatumGetInt64() rather than comparing a Datum to 0, plus the two new
Asserts. (The existing code is correct, since hasLimitedFrame carries
the limited/unbounded distinction separately, but the refactor is the
clearer shape.) I'll take it.

One thing to watch when dropping hasLimitedFrame, though: a zero offset
is still a valid limited frame -- ROWS BETWEEN CURRENT ROW AND CURRENT
ROW, and ... AND 0 FOLLOWING -- so the new check must not treat offset 0
as unbounded, otherwise the boundary check is skipped and a quantified
pattern (A+) silently absorbs the whole partition. The limited/unbounded
distinction lives in the frame-option flag, not the offset value, so the
refactor has to keep that flag in the picture.

We don't currently cover the offset-0 quantified case -- the two
zero-offset frame tests in rpr_base both use a plain PATTERN (A) -- so
I'll add an A+ regression test for it.


> [nfa_evaluate_row's window_gettupleslot returning false]
> indicates that it's not possible for (currentPos > ctxFrameEnd).
> therefore [if (currentPos >= ctxFrameEnd) ...]
> should change to ``if (currentPos == ctxFrameEnd)``.
> With that, the comment wording "exceeded" seems wrong too.

Agreed. `>` is unreachable by the loop invariant -- currentPos advances
by exactly one per row, and once a context is finalized the
states == NULL guard skips it -- so >= and == behave identically here,
and == states the intent better. The `>=` was a defensive guard against
an overshoot that can't actually happen; v48 tightens it to ==, changes
the comment from "exceeded" to "reached", and moves that defense into an
Assert(currentPos <= ctxFrameEnd) so a future change that breaks the
invariant trips immediately instead of silently.


> Since I couldn't generate a coverage report, I am using elog(INFO) to
> test reachability. Rerun the regress tests, you will find out that the
> ELSE IF branch below is not reached by current regress tests.
>     else if (targetCtx->states == NULL)
>     {
>         /* Context already completed - skip to result registration */
>         goto register_result;
>         elog(INFO, "XXXX reached");
>     }

The elog in your snippet sits *after* the goto, so it can never print
even though the branch runs. Moving it above the goto shows it firing
across the suite. Easy mistake; I've put a probe right after a
goto/return more times than I'd like to admit.

The branch is correct as written: under SKIP TO NEXT ROW an overlapping
context can finish (and be preserved by cleanup, since
matchEndRow >= matchStartRow) before the next update_reduced_frame()
call lands on its start row. So no new test is needed -- I'll just add a
comment above it explaining why the head context can already be complete
there.


Summary of decisions, in the order above:


  RPRContext consolidation         Tatsuo's call -- non-blocking for v48
  get_reduced_frame_status order   Keep -- cascade idiom; add comments
  README terminology + typo        Accept -- definitions block + rename
  ExecRPRProcessRow refactor       Accept -- but fix frameOffset
  currentPos >= -> ==              Accept -- plus comment + Assert
  states == NULL branch coverage   Already reached (155x) -- add comment


I'll post an incremental patch shortly with the accepted changes and the
comment additions; the RPRContext question is for Tatsuo.

Thanks again,
Henson

  340. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-05-29T09:41:52Z

    Hi Jian, Tatsuo,

Jian, your review keeps paying off beyond the items themselves.  Working
through the ExecRPRProcessRow refactor you proposed -- dropping
hasLimitedFrame and fixing the "endOffsetValue != 0" Datum-vs-0
comparison -- and writing the offset-0 regression test it calls for sent
me back to the standard, where I hit a conformance reading I'd value both
of your judgments on.

The case is the single-row full window frame.  It has two spellings,
semantically identical -- both make the full window frame exactly the
current row:

    ROWS BETWEEN CURRENT ROW AND CURRENT ROW
    ROWS BETWEEN CURRENT ROW AND 0 FOLLOWING

PostgreSQL accepts both today (rpr_base tests the 0 FOLLOWING form with
PATTERN (A)).  But I'm no longer sure either is sanctioned by Subclause
6.10.2, "ROWS BETWEEN CURRENT ROW AND", which reads (paraphrasing
ISO/IEC 19075-5):

  "When performing row pattern recognition in a window, only two options
   are allowed for specifying the window frame extent:
     - ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
     - ROWS BETWEEN CURRENT ROW AND offset FOLLOWING ... [the offset]
       shall be a positive integer ..."

Two things put the single-row frame outside a literal reading, and they
hit both spellings equally:

  - CURRENT ROW as the *end* bound is not one of the two listed options;
    and

  - the offset form requires a *positive integer*, and 0 is not positive
    -- so AND 0 FOLLOWING is no better sanctioned than AND CURRENT ROW.

The two spellings therefore stand or fall together: they denote the same
one-row frame, and neither is literally among 6.10.2's options.
(README.rpr X-4 already follows the strict line -- it lists only
"UNBOUNDED FOLLOWING or n FOLLOWING".)

Either way the behavior is unambiguous: a single-row search space, so the
pattern can only ever map the current row.  That makes it a degenerate,
WHERE-like use rather than real row-pattern matching -- which one might
read as benign (so allow) or as pointless (so reject), so it doesn't
settle the question for me.

This is, in effect, a borderline, harmless standard violation: it sits
just outside 6.10.2 as written, yet is well-defined, runs cleanly, and
leaves every conforming frame unchanged.

So, concretely, the choice:

  (a) Allow it -- accept both spellings and document the degenerate
      frame as a deliberate extension; or

  (b) Forbid it -- follow 6.10.2 to the letter ("only two options" and a
      "positive integer" offset) and reject both spellings in
      transformRPR().

Which reading do you take?  Tatsuo, as co-author your call carries it;
Jian, I'd value yours too, since you were just in this code.

This isn't blocking.  The implementation accepts offset 0 today, so for
now my regression test exercises it (via the 0 FOLLOWING spelling, as
rpr_base already does); if the answer is (b), that test and the existing
one become error tests together.

Thanks,
Henson

  341. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-05-30T02:47:37Z

    Hi Henson,

> Hi Jian, Tatsuo,
> 
> Jian, your review keeps paying off beyond the items themselves.  Working
> through the ExecRPRProcessRow refactor you proposed -- dropping
> hasLimitedFrame and fixing the "endOffsetValue != 0" Datum-vs-0
> comparison -- and writing the offset-0 regression test it calls for sent
> me back to the standard, where I hit a conformance reading I'd value both
> of your judgments on.
> 
> The case is the single-row full window frame.  It has two spellings,
> semantically identical -- both make the full window frame exactly the
> current row:
> 
>     ROWS BETWEEN CURRENT ROW AND CURRENT ROW
>     ROWS BETWEEN CURRENT ROW AND 0 FOLLOWING
> 
> PostgreSQL accepts both today (rpr_base tests the 0 FOLLOWING form with
> PATTERN (A)).  But I'm no longer sure either is sanctioned by Subclause
> 6.10.2, "ROWS BETWEEN CURRENT ROW AND", which reads (paraphrasing
> ISO/IEC 19075-5):
> 
>   "When performing row pattern recognition in a window, only two options
>    are allowed for specifying the window frame extent:
>      - ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>      - ROWS BETWEEN CURRENT ROW AND offset FOLLOWING ... [the offset]
>        shall be a positive integer ..."
> 
> Two things put the single-row frame outside a literal reading, and they
> hit both spellings equally:
> 
>   - CURRENT ROW as the *end* bound is not one of the two listed options;
>     and
> 
>   - the offset form requires a *positive integer*, and 0 is not positive
>     -- so AND 0 FOLLOWING is no better sanctioned than AND CURRENT ROW.
> 
> The two spellings therefore stand or fall together: they denote the same
> one-row frame, and neither is literally among 6.10.2's options.
> (README.rpr X-4 already follows the strict line -- it lists only
> "UNBOUNDED FOLLOWING or n FOLLOWING".)
> 
> Either way the behavior is unambiguous: a single-row search space, so the
> pattern can only ever map the current row.  That makes it a degenerate,
> WHERE-like use rather than real row-pattern matching -- which one might
> read as benign (so allow) or as pointless (so reject), so it doesn't
> settle the question for me.
> 
> This is, in effect, a borderline, harmless standard violation: it sits
> just outside 6.10.2 as written, yet is well-defined, runs cleanly, and
> leaves every conforming frame unchanged.
> 
> So, concretely, the choice:
> 
>   (a) Allow it -- accept both spellings and document the degenerate
>       frame as a deliberate extension; or
> 
>   (b) Forbid it -- follow 6.10.2 to the letter ("only two options" and a
>       "positive integer" offset) and reject both spellings in
>       transformRPR().
> 
> Which reading do you take?  Tatsuo, as co-author your call carries it;
> Jian, I'd value yours too, since you were just in this code.
> 
> This isn't blocking.  The implementation accepts offset 0 today, so for
> now my regression test exercises it (via the 0 FOLLOWING spelling, as
> rpr_base already does); if the answer is (b), that test and the existing
> one become error tests together.

I vote for (b).

Reason 1: (a) does not give clear benefit to users.

Reason 2: we haven't implemented some large features yet (MEASURES
etc.)  If we follow (a), we may have to add more code to support it
in the process of implementing MEASURES when they are not necessary if
we had chosen (b).

Reason 3: In the future, we may find an optimization if 0 FOLLOWING is
prohibited.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  342. Re: Row pattern recognition

    jian he <jian.universality@gmail.com> — 2026-05-30T06:45:25Z

    Hi.

-- Consecutive VAR merge: A A+ -> a{2,}
-- Tests line 251: child->max == RPR_QUANTITY_INF branch in mergeConsecutiveVars
-- prev: A{1,1} (finite), child: A+ (infinite) triggers line 251 evaluation
EXPLAIN (COSTS OFF)
SELECT COUNT(*) OVER w FROM rpr_plan
WINDOW w AS (ORDER BY id ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
             PATTERN (A A+) DEFINE A AS val > 0);

-- Consecutive VAR merge: A A+ -> a{2,}
-- Tests line 251: child->max == RPR_QUANTITY_INF branch in mergeConsecutiveVars
-- prev: A{1,1} (finite), child: A+ (infinite) triggers line 251 evaluation
EXPLAIN (COSTS OFF)
SELECT COUNT(*) OVER w FROM rpr_plan
WINDOW w AS (ORDER BY id ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
             PATTERN (A A+) DEFINE A AS val > 0);

Comments like "Tests line 251" should not appear in SQL, since the
line number would change.
So, we need to rephrase this comment. Please don't delete it, it may
clarify the patch for future readers.


/*
 * nfa_add_state_unique
 *
 * Add a state to ctx->states at the END, only if no duplicate exists.
 * Returns true if state was added, false if duplicate found (state is freed).
 * Earlier states have better lexical order (DFS traversal order), so
existing wins.
 */
static bool
nfa_add_state_unique(WindowAggState *winstate, RPRNFAContext *ctx,
RPRNFAState *state)

The "at the END" confuses me. It may also refers to RPR_VARID_END /
RPRElemIsEnd.
How about something like:
"Add the state to the end of the ctx->states linked list, but only if
a duplicate state is not already present."

I think a bunch of these uppercase ENDs should just be lowercase "end",
capitalizing them may collides visually with RPR_VARID_END and RPRElemIsEnd and
makes the comments harder to read.

In nfa_match:
"Non-VAR elements (ALT, END, FIN) are kept as-is for advance phase."

Here END does mean RPRElemIsEnd, right? That one's probably fine as uppercase
since it's listed alongside other element kinds.

But then:
/*
 * Evaluate VAR elements against current row. For VARs that reach max
 * count with END next, advance through END chain inline so absorb phase
 * can compare states at judgment points.
 */
I'm think END here means the tail of the RPRNFAContext->RPRNFAState linked
list, not the element kind -- which is exactly the ambiguity I'm worried about.

Could we lowercase the list-end ones and keep uppercase only when actually
referring to the END element kind?


The comment under nfa_advance_var is confusing (for me).
/* After a successful match, count >= 1, so at least one must be true */

nfa_advance_var doesn't actually know anything about match status (i
think), it can be reached with count == 0.
I added this elog(INFO) to confirm:
    if (count < 1 && currentPos > -1)
        elog(INFO, "should not reach, count is %d, currentPos=%ld",
count, currentPos);

...and it does fire, so the comment's premise isn't quite right.

------------------------------------------------
please check the attached minor refactoring.
1.  Refactor struct WindowAggState: Remove the nfaStateSize and nfaVisitedNWords
fields because they are constant, and we can easily compute it, seems
not necessary to stay in WindowAggState.

2. Rename nfa_context_alloc() to nfa_context_make(), nfa_state_alloc()
to nfa_state_make(), nfa_state_create() to nfa_state_clone().
Rationale: We have makeNode, changing "alloc" to "make" would be more
intuitive, IMHO.
In tablecmd.c, we have CloneForeignKeyConstraints, which is based on
existing information, creating a new node, here we are doing the same
in
nfa_state_create.

3. Refactor functions: nfa_advance_alt, nfa_advance_begin,
nfa_advance_end, and nfa_advance_var.
Because the (RPRPatternElement *elem) parameter is unnecessary.



--
jian
https://www.enterprisedb.com/

  343. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-05-30T13:45:39Z

    [mailing-list reply draft -- pgsql-hackers, RPR thread.
 In-reply-to Jian's round-3 review, 2026-05-30 06:45
 (Message-ID CACJufxEsaU8GQ4yeXTWhAO8VjbrZTh5CpvUqz=
4a3T0Cwz44pA@mail.gmail.com).
 Covers R3-1..R3-4 (comments) and the three attached refactors R3-5a/b/c.
 All accepted; fold into the v48 refactor series.]

Subject: Re: Row pattern recognition

Hi Jian,

Thanks for the round-3 pass -- all seven are good catches. Going through
them inline below, with a summary of decisions at the end.

> Comments like "Tests line 251" should not appear in SQL, since the
> line number would change.
> So, we need to rephrase this comment. Please don't delete it, it may
> clarify the patch for future readers.

Agreed on both counts -- the intent (this case exercises the
child->max == RPR_QUANTITY_INF branch in mergeConsecutiveVars, where a
finite prev meets an infinite child) is worth keeping; only the "line 251"
anchor is fragile. I'll rephrase it to name the branch and the condition
it covers instead of the line, so it survives any later shuffle.

There's a parallel one a few tests down -- "Tests line 325 ... in
mergeConsecutiveGroups" -- with the same problem, so I'll fix both
together on the same principle.

> The "at the END" confuses me. It may also refers to RPR_VARID_END /
> RPRElemIsEnd.
> How about something like:
> "Add the state to the end of the ctx->states linked list, but only if
> a duplicate state is not already present."

Your wording is clearer than mine -- I'll take it almost verbatim. "at the
END" there meant the list tail, and capitalizing it was exactly the wrong
choice next to the END element kind.

> I think a bunch of these uppercase ENDs should just be lowercase "end",
> [...]
> Could we lowercase the list-end ones and keep uppercase only when actually
> referring to the END element kind?

Yes -- that's the right rule, and I'll go through them on it: lowercase
"end" for the linked-list tail, uppercase END only when it means the
element kind (RPRElemIsEnd / RPR_VARID_END). On the two you singled out:

  - nfa_match, "Non-VAR elements (ALT, END, FIN) ..." -- element kind,
    listed with the others, stays uppercase. Agreed.

  - "advance through END chain inline ..." -- this one is actually the
    element kind too: it walks the chain of group-END *elements* up to the
    absorption judgment point, not the state list. So it stays uppercase,
    but since it tripped you up I'll reword it to "chain of END elements"
    to make that unambiguous rather than leaning on the reader to infer it.

> The comment under nfa_advance_var is confusing (for me).
> /* After a successful match, count >= 1, so at least one must be true */
> nfa_advance_var doesn't actually know anything about match status (i
> think), it can be reached with count == 0.
> [...] and it does fire, so the comment's premise isn't quite right.

You're right, and thanks for confirming it with the elog. nfa_advance_var
is part of the advance phase, which generates every reachable next state
regardless of what matched -- so it's routinely entered with count == 0,
for a VAR just reached through an ALT branch or a group that hasn't been
evaluated yet. The comment carries match-phase reasoning into a phase that
has no notion of matching. The Assert above still holds, but on the
structure of a VAR's bounds, not on any prior match, so I'll reword the
comment to say that instead.

> please check the attached minor refactoring.
> 1.  Refactor struct WindowAggState: Remove the nfaStateSize and
>     nfaVisitedNWords fields because they are constant, and we can easily
>     compute it, [...] not necessary to stay in WindowAggState.
> 2. Rename nfa_context_alloc() to nfa_context_make(), nfa_state_alloc()
>    to nfa_state_make(), nfa_state_create() to nfa_state_clone().
>    Rationale: We have makeNode [...]. In tablecmd.c, we have
>    CloneForeignKeyConstraints, which is based on existing information,
>    creating a new node, here we are doing the same in nfa_state_create.
> 3. Refactor functions: nfa_advance_alt, nfa_advance_begin,
>    nfa_advance_end, and nfa_advance_var. Because the
>    (RPRPatternElement *elem) parameter is unnecessary.

Looked at all three. 0002 and 0003 I'll take; 0001 I'd split.

  1. The two fields differ. nfaVisitedNWords -- yes: in the current tree
     it's read only once, at init, to size the visited bitmap, since the
     per-row reset clears just the high-water [min,max] range. Demoting it
     to a local is clean. (Heads up: the patch is on v47, before that
     high-water-mark change, so its nfa_advance hunk no longer applies and
     drops out on rebase.)

     nfaStateSize I'd keep. The patch recomputes it inside the allocator,
     i.e. on every state allocation -- the hottest path in the engine; a
     pathological (V1|...|Vk)* can allocate hundreds of millions of states.
     It's a loop-invariant fixed at init, so I'd rather keep it precomputed
     than rebuild offsetof + maxDepth*4 each alloc -- the same reasoning I
     gave for keeping the frame offsets out of the per-row loop.

  2. make/clone: agreed, and there's a closer precedent than I'd have
     reached for -- our own regex NFA engine already clones states
     (clonesuccessorstates / cloneouts in regcomp.c), so clone is the right
     word for duplicating an NFA state, not just the CloneForeignKey sense.
     The pair reads well too: make for the blank allocation, clone for
     building a state from an existing one's counts. (create was itself an
     earlier rename of mine from clone "for clarity," so this just lands us
     back on clone, which I now agree is better.) When I apply it I'll also
     tidy two comments the rename left behind -- the nfa_state_make header
     now reads "a new RPRNFAState state," and nfa_state_clone's body comment
     still says "Create."

  3. elem parameter: here I'd keep it, on the same grounds as nfaStateSize.
     The four functions have a single caller, nfa_advance_state, which
     already computes elem = &elements[state->elemIdx] for its own
     mark-visited check and switch, and hands it down. Dropping the
     parameter makes each function rebuild that same address on the
     per-dispatch advance path -- discarding a value the caller already
     holds. And with one trusted caller, the mismatch the change guards
     against can't really arise. The cost either way is tiny; I'd just
     rather keep the value flowing from where it's already computed than
     recompute it, which is the same call I made on nfaStateSize.

Summary of decisions, in the order above:

  "Tests line N" test comments     Rephrase -- name the branch, not the
line (both)
  "at the END" wording             Accept -- your wording, near-verbatim
  uppercase END cleanup            Accept -- lowercase the list-end ones,
keep the kind
  nfa_advance_var count comment    Reword -- wrong premise; state the
structural reason
  0001 nfaVisitedNWords            Accept -- demote to a local
  0001 nfaStateSize                Keep -- hottest path; leave it
precomputed
  0002 make / clone rename         Accept -- plus tidy two stray comments
  0003 drop the elem parameter     Keep -- single caller already holds it

So the next revision carries the four comment fixes, 0002, and the
nfaVisitedNWords half of 0001; nfaStateSize and the elem parameter I'd keep
as they are. Happy to be talked round on either if you see a cost I'm
missing.

Thanks again,
Henson

  344. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-05-30T14:08:38Z

    Hi all,

Resolved since the last post:

  D1. Single-row frame conformance, Subclause 6.10.2 -- Tatsuo's call [6]
      was to reject. Both ROWS BETWEEN CURRENT ROW AND CURRENT ROW and
      AND 0 FOLLOWING are now rejected (nocfbot-0025), which in turn
      unblocks the held ExecRPRProcessRow change (nocfbot-0026).

Open decisions (repeated with context at the end):

  D2. The RPRContext consolidation -- Tatsuo's call as co-author;
      non-blocking either way [4].
  D3. The AST "absorption" rename -- Tatsuo's call [2].

  [1] Jian's review, round 1 (2026-05-26):

https://postgr.es/m/CACJufxH-DZePhbdJM=8nNYceQiSbbXXLTw54iLhxiynQ+4hbBA@mail.gmail.com
  [2] my round-1 reply -- AST "absorption" rename deferred to Tatsuo (D3,
2026-05-27):

https://postgr.es/m/CAAAe_zDephfiDA_A3FN0hCymJRogEr=Rt3QoCTf4qMYDLk+xNA@mail.gmail.com
  [3] Jian's review, round 2 (2026-05-28):

https://postgr.es/m/CACJufxGX17thWuEOq1tM5xbRHz2HXm1asooZC3GV25MYGmYqLQ@mail.gmail.com
  [4] my round-2 reply -- RPRContext deferred to Tatsuo (D2, 2026-05-29):

https://postgr.es/m/CAAAe_zAH0MvP0TBmW3PTLeHjEpiyBz0473zJRM9pwLpseefMNw@mail.gmail.com
  [5] single-row frame conformance question (D1, 2026-05-29):

https://postgr.es/m/CAAAe_zCbSU=dd-4qTL2QaBQwQ-cf51N_851a9Y5rOoz0wj0aXw@mail.gmail.com
  [6] Tatsuo's reply -- reject single-row frames (D1 resolved, 2026-05-30):

https://postgr.es/m/20260530.114737.1416684464524168377.ishii@postgresql.org
  [7] Jian's review, round 3 (2026-05-30):

https://postgr.es/m/CACJufxEsaU8GQ4yeXTWhAO8VjbrZTh5CpvUqz=4a3T0Cwz44pA@mail.gmail.com

Attached: the v47 feature series (v47-0001..0009) rebased onto current
master, plus the incremental patch series carried on top of it.

Base:

  9a41b34a287  2026-05-26  doc:  add comma to UPDATE docs, for consistency

Unchanged -- rebase only (already posted; only rebased, no content change).
Titles for reference:

  nocfbot-0001  Add DEFINE non-volatile baseline to rpr_integration B9
  nocfbot-0002  Unify RPR DEFINE walkers and reject volatile callees
  nocfbot-0003  Cover RPR empty-match path with EXPLAIN tests; fix stale
XXX comments
  nocfbot-0004  Reclassify DEFINE qualifier check and reword diagnostic to
"expression"
  nocfbot-0005  Sync stale comments on DEFINE/PATTERN handling
  nocfbot-0006  Add trailing commas to RPR enum definitions
  nocfbot-0007  Remove optional outer parentheses from ereport() calls in
RPR files
  nocfbot-0008  Add high-water mark tracking to NFA visited bitmap reset
  nocfbot-0009  Document DEFINE subquery rejection as intentional
over-rejection
  nocfbot-0010  Remove duplicate #include in nodeWindowAgg.c
  nocfbot-0011  Normalize SQL/RPR standard references
  nocfbot-0012  Add rpr_integration B7 cases for RPR in recursive query
  nocfbot-0013  Reject row pattern recognition in recursive queries
  nocfbot-0014  Enhance README.rpr per Tatsuo Ishii's review
  nocfbot-0015  Round out README.rpr WindowAggState field coverage
  nocfbot-0016  Add raw_expression_tree_walker coverage for RPR raw nodes

  (nocfbot-0016 was sent earlier as 0015; renumbered here so the review
  series runs contiguously from 0017.)

New incremental patches -- nocfbot-0017 onward. These apply Jian He's
review (rounds 1 [1] and 2 [3]) and settle D1. nocfbot-0017..0024 and
0026 are comment / doc / test plus a signature change and a couple of
Assert additions -- no behavior change. nocfbot-0025 is the one
user-visible change: it rejects the single-row frame per D1 [6].

  nocfbot-0017  Enhance README.rpr
      Chapter VIII absorption intro + a worked PATTERN (A+) trace;
      "Depth-First Search" spelled out at first use; the stale
      nfa_advance(initialAdvance=...) reference replaced.

  nocfbot-0018  Clarify execRPR.c comments and tighten an Assert
      Document the NFA invariants (compareDepth slot arithmetic, the
      asymmetric visited-marking scheme, the greedy/non-nullable BEGIN
      label, the ALT depth break, the state->next reset boundary),
      reframe ExecRPRFinalizeAllContexts as the partition-end policy
      holder, and add a defensive Assert in nfa_advance_var.

  nocfbot-0019  nfa_add_state_unique: bool return -> void
      The return value was unused.

  nocfbot-0020  Reluctant bounded mid-band test (rpr_nfa)
      A{3,5}? B, which drives the VAR-level count in nfa_advance_var
      through 3..5.

  nocfbot-0021  Define RPR absorption terminology in README.rpr
      Terminology block for the "match_start dep." column (none /
      direct / boundary check), the "boundary chk" typo fix, the
      count-dominance definition, and the match_start_dependent rename.

  nocfbot-0022  Document the get_reduced_frame_status cascade invariant
      The branches form an order-dependent early-return cascade; the
      running invariant is spelled out so the order reads as intentional.

  nocfbot-0023  Explain the completed-head-context branch in
                update_reduced_frame
      Why a head context can already be complete under SKIP TO NEXT ROW.

  nocfbot-0024  Tighten the frame-boundary check from >= to ==
      The > case is unreachable by the loop invariant; the defense moves
      into an Assert(currentPos <= ctxFrameEnd), and the comment changes
      from "exceeded" to "reached".

  nocfbot-0025  Reject single-row window frame in row pattern recognition
      Per D1 [6], the frame end must be UNBOUNDED FOLLOWING or a positive
      offset FOLLOWING. CURRENT ROW is rejected in transformRPR() at parse
      time; a zero offset -- which need not be a constant -- in
      calculate_frame_offsets() at run time. The two single-row rpr_base
      tests become error cases, with a bind-parameter error test added.

  nocfbot-0026  Remove the redundant zero check on the RPR frame ending
                offset
      With the single-row frame now rejected, a limited frame always
      carries a real offset, so the "endOffsetValue != 0" guard -- which
      compared a Datum directly to zero -- is dropped, leaving a plain
      DatumGetInt64(). This is the offset half of Jian's ExecRPRProcessRow
      cleanup; the structural half (dropping the hasLimitedFrame
      parameter) I've left as-is -- it is loop-invariant and computed once
      outside the per-row loop, so moving it into ExecRPRProcessRow would
      just repeat the work each row.

Coverage -- my decision summaries [2], [4], with the patch each became:

  Round 1 [2]
    Short-circuit optimization       Separate series   -> separate series
    Absorption README narrative      Accept            -> nocfbot-0017
    AST-level "absorption" rename    Pending Tatsuo    -> D3
    DFS expansion                    Accept            -> nocfbot-0017
    initialAdvance README mismatch   Accept            -> nocfbot-0017
    Defensive Assert in advance_var  Accept            -> nocfbot-0018
    Finalize unnecessary?            Keep              -> nocfbot-0018
    Greedy comment label             Accept            -> nocfbot-0018
    state->next reset                Decline           -> nocfbot-0018
    count >= 3 test coverage         Accept            -> nocfbot-0020
    visited marking purpose          Accept            -> nocfbot-0018
    compareDepth comment             Accept            -> nocfbot-0018
    Unused bool return               Accept            -> nocfbot-0019
    ALT depth invariant Assert       Decline           -> nocfbot-0018

  Round 2 [4]
    RPRContext consolidation         Tatsuo's call     -> D2
    get_reduced_frame_status order   Keep              -> nocfbot-0022
    README terminology + typo        Accept            -> nocfbot-0021
    ExecRPRProcessRow refactor       Datum fix only    -> nocfbot-0026
    single-row frame (6.10.2, D1)    Reject (Tatsuo)   -> nocfbot-0025
    currentPos >= -> ==              Accept            -> nocfbot-0024
    states == NULL branch coverage   Already reached   -> nocfbot-0023

Remaining work and decisions

  Decisions (need your input):
    D2  RPRContext consolidation -- Tatsuo.
    D3  AST "absorption" rename -- Tatsuo.

  Held pending a decision:
    - the RPRContext consolidation itself -- done if D2 says go.
    - the AST "absorption" rename itself -- done if D3 says go.

  (D1 settled: the parse-time frame-end check, the offset-0 test as an
   error case, and the Datum-comparison fix are now in nocfbot-0025/0026.)

  From Jian's round-3 review [7] (into the next revision):
    - four comment fixes: the line-number test anchors, the "at the END"
      and uppercase-END wording, and the nfa_advance_var count premise.
    - the make/clone rename of the NFA helpers (one of three attached
      refactors); the other two -- dropping nfaStateSize and the elem
      parameter -- I'd keep, both on hot-path grounds.

  From our in-house review (separate follow-ups):
    - a quantifier-normalization correctness fix (nested unbounded
      quantifiers such as (A{2,})* are currently mis-normalized)
    - a per-tuple memory-context fix in DEFINE evaluation (still verifying)
    - smaller correctness/conformance fixes (an overflow guard, a few
      missing parse-time checks, EXPLAIN output details)
    - documentation gaps (comments, README, SGML)
    - added regression tests (round-trip deparse, edge-case offsets)

  Before the v48 fold (from Jian's off-list comments):
    - INT_MAX -> PG_INT32_MAX (the unbounded-quantifier sentinel; ~24 sites)
    - foreach + lfirst() -> foreach_node (~33 sites)
    - foreach_current_index, dropping the redundant break (3 sites)

  Work, no decision needed:
    - short-circuit (lazy eval) -- stop evaluating a DEFINE predicate
      once its outcome is fixed. A separate series.
      It turns on a standard-interpretation point -- whether skipping is
      sound when a dropped subexpression has side effects.

Thanks again to Jian for the careful reading.

Henson

  345. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-05-31T00:11:19Z

    Hi Henson, Jian,

>> In src/include/nodes/execnodes.h, we're adding quite a few fields to
>> WindowAggState that are only used for RPR queries.
>> Should we consolidate these fields behind a single pointer (named
>> RPRContext) to keep the WindowAggState size smaller for non-RPR
>> queries?
> 
> The size win is real -- it's roughly 450-500 bytes per WindowAggState
> that every non-RPR window query carries today. But it takes a wide
> code change to get there: it reshapes every RPR access path, a
> sizable (if mostly mechanical) diff across nodeWindowAgg.c,
> execRPR.c, and the explain side.
> 
> Tatsuo, as co-author -- do you want this in v48? If you do, I'll
> prepare the RPRContext consolidation as an incremental patch for you to
> fold into v48. It isn't blocking either way.

Basically I think Jian's idea is good. In addition to the size reason
above, we would have less code changes when we adapt existing R020
codes to R010.

However it will need a wide code change as Henson said. I would like
to focus on stabilizing our code for now. Therefore I would not want
the refactoring in v48.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  346. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-05-31T01:32:58Z

    Hi Henson,

>> For src/backend/executor/README.rpr:
>> We should explicitly explain 'absorbable' and 'absorption' somewhere in
>> README.rpr, as the text currently just assumes the reader knows what they
> mean.
>> Using some example illustrate "absorption" meaning, put it on
>> README.rpr would be great.
>> We can also mention that 'DFS' refers to Depth-First Search".
> 
> Acknowledged, and the request surfaced an underlying problem in the
> README's terminology. "Absorption" is currently used for two
> distinct things: an AST-level rewrite in Phase 1 that pulls
> identical sequences around a group inside it, and the runtime
> context-equivalence collapse that drives the O(n^2) -> O(n)
> optimization. Sharing the word leaves a reader encountering
> "absorbable" early on without an anchor.
> 
> Rather than disambiguate by qualifier ("prefix/suffix absorption"
> vs "context absorption"), I'd lean toward renaming the AST-level
> case so "absorption" stays reserved for the runtime concept. The
> README then only needs to explain absorption in one place, in
> detail, without the disambig preamble.
> 
> For the rename, "prefix/suffix merging" feels like the natural fit
> -- the other AST-level optimizations in the same Phase 1 are already
> named "consecutive variable / group / ALT merging", so it slots in
> cleanly. "Prefix/suffix factoring" is another candidate if a more
> descriptive verb is preferred.
> 
> Tatsuo, curious what you think of this direction and naming. Happy
> to take any name you prefer for the AST-level operation, or to keep
> the original "absorption" wording with stronger forward-references
> if you'd rather not rename.

Although I don't have any particular strong preferences, keeping
"absorption" for the runtime concept sounds good to me.

For AST level name changing, "prefix/suffix merging" seems to be
already used in other areas according to Google: LLM, Linker, and
string manipulation in DNA. In the normal expression engine area, it
looks like "flattening nested quantifiers" or "quantifiers reduction"
are used for the case. So, for example, "prefix/suffix quantifiers
reduction" seems to be more appropriate?  (If you don't mind it's too
long) In any case, I would like to respect your opinion.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  347. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-05-31T12:33:13Z

    Hi Henson,

> Attached is the v47 patches for Row pattern recognition (SQL/RPR).

I accidentaly noticed that v47-0002 changes findTargetlistEntrySQL99
from static to extern.

-static TargetEntry *findTargetlistEntrySQL99(ParseState *pstate, Node *node,
-											 List **tlist, ParseExprKind exprKind);

I think this is not necessary anymore since findTargetlistEntrySQL99
is not used outside parse_clause.c.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  348. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-01T00:40:40Z

    Hi Tatsuo, Jian,

Thanks, Tatsuo -- your two notes settle both open questions. Answering both
inline below, with a note at the end on the follow-up work.

> Basically I think Jian's idea is good. In addition to the size reason
> above, we would have less code changes when we adapt existing R020
> codes to R010.
>
> However it will need a wide code change as Henson said. I would like
> to focus on stabilizing our code for now. Therefore I would not want
> the refactoring in v48.

Agreed -- out of v48, stabilize first. On R010, I'd treat it as the design
lens, not the schedule: it's far out, so rather than hold RPRContext back
until then, I'd do the consolidation at a sensible point after v48 but shape
it against R010 (what is shared versus per-context, how the fields group),
so
one engine core can later back both the R020 window path and R010 without a
second reshape. Not v48 -- on its own schedule once we're stable.

I'll admit the R010 connection had been nagging at me for a while without a
clean answer, and Jian's consolidation suggestion turns out to land right on
it: framed as a size win, it's really the structural move that opens the
R010
path -- that reframing is his. Which is why I'd rather shape it
deliberately,
as the shared engine core, than fold it in as churn.

> Although I don't have any particular strong preferences, keeping
> "absorption" for the runtime concept sounds good to me.

Good -- "absorption" stays reserved for the runtime context-equivalence
collapse, and the README explains it in one place.

> For AST level name changing, "prefix/suffix merging" seems to be
> already used in other areas according to Google: LLM, Linker, and
> string manipulation in DNA. In the normal expression engine area, it
> looks like "flattening nested quantifiers" or "quantifiers reduction"
> are used for the case. So, for example, "prefix/suffix quantifiers
> reduction" seems to be more appropriate?  (If you don't mind it's too
> long) In any case, I would like to respect your opinion.

Thanks -- you're right that "merging" is well-worn elsewhere, and I'll be
honest that "prefix/suffix merging" isn't a term I'd defend on the merits.
Keeping it for v48 is really a stopgap to contain the ripple: the sibling
Phase-1 rewrites are already named "consecutive variable / group / ALT
merging", so switching to the "flattening / reduction" family would force
renaming those too for consistency. So I'd treat the term itself as
genuinely
open -- your "flattening / reduction" neighborhood is the right one -- and
converge on the established academic naming as the paper I'm preparing on
the
algorithm, together with a university research group, takes shape. For now
I'd
ship "prefix/suffix merging" only to keep the README internally consistent,
and fold the settled term into the glossary pass once the paper lands on
it. A
doc-level name is cheap to revise later.

Both of these -- the RPRContext reshape and the naming/terminology -- are
"after v48, by discussion" items, and they aren't the only ones. Rather than
pin down the scope of that follow-up now, I'd suggest we pick it up together
once v48 is stable.

Thanks again,
Henson

  349. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-01T01:16:54Z

    Hi Tatsuo,

> I accidentaly noticed that v47-0002 changes findTargetlistEntrySQL99
> from static to extern.
> [...]
> I think this is not necessary anymore since findTargetlistEntrySQL99
> is not used outside parse_clause.c.

Good catch -- you're right, and I'll revert it to static in v48. I
double-checked the current tree: the only callers left are inside
parse_clause.c (the findTargetlistEntry wrapper, plus the GROUP BY and ORDER
BY paths), so the extern prototype in parse_clause.h is now unused. The
revert is just dropping that prototype and restoring the static qualifier
with the in-file forward declaration it used to have.

It's worth saying why it went extern in the first place, since the reason is
no longer visible in the tree:

The DEFINE clause needs its referenced columns present in the plan's
targetlist to be evaluable at run time. The original implementation did
that from the RPR side, in parse_rpr.c, by calling
findTargetlistEntrySQL99()
with resjunk = true to add the missing entry -- and since that function was
static in parse_clause.c, reaching it across files is what required exposing
it as extern.

That approach added the whole DEFINE expression to the targetlist, and that
turned out to be the source of a SIGSEGV: when an RPR window and a plain
window coexist, the non-RPR WindowAgg inherited targetlist entries carrying
RPRNavExpr nodes it has no way to evaluate. The fix was to add only the Vars
a DEFINE references (with a guard in allpaths.c to keep those columns from
being pruned), and that is what removed the cross-file call. So the extern
has simply outlived its caller -- exactly as you spotted.

The one loose end there is an optimization, not a correctness issue: the
allpaths.c guard is deliberately coarse, so it also blocks removing a
WindowAgg whose RPR WindowFuncs are all unused. Doing that precisely means
restructuring remove_unused_subquery_outputs(), which runs for every
subquery and not just RPR -- broad enough that I'm treating it as a
longer-term item rather than part of this work. It doesn't bring the
external call back -- the Var-only path stays -- so the revert to static is
safe independently of it.

I'll fold the static revert into v48.

Thanks,
Henson

  350. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-01T02:11:19Z

    Hi Henson,

> Good catch -- you're right, and I'll revert it to static in v48. I
> double-checked the current tree: the only callers left are inside
> parse_clause.c (the findTargetlistEntry wrapper, plus the GROUP BY and ORDER
> BY paths), so the extern prototype in parse_clause.h is now unused. The
> revert is just dropping that prototype and restoring the static qualifier
> with the in-file forward declaration it used to have.
> 
> It's worth saying why it went extern in the first place, since the reason is
> no longer visible in the tree:
> 
> The DEFINE clause needs its referenced columns present in the plan's
> targetlist to be evaluable at run time. The original implementation did
> that from the RPR side, in parse_rpr.c, by calling
> findTargetlistEntrySQL99()
> with resjunk = true to add the missing entry -- and since that function was
> static in parse_clause.c, reaching it across files is what required exposing
> it as extern.
> 
> That approach added the whole DEFINE expression to the targetlist, and that
> turned out to be the source of a SIGSEGV: when an RPR window and a plain
> window coexist, the non-RPR WindowAgg inherited targetlist entries carrying
> RPRNavExpr nodes it has no way to evaluate. The fix was to add only the Vars
> a DEFINE references (with a guard in allpaths.c to keep those columns from
> being pruned), and that is what removed the cross-file call. So the extern
> has simply outlived its caller -- exactly as you spotted.
> 
> The one loose end there is an optimization, not a correctness issue: the
> allpaths.c guard is deliberately coarse, so it also blocks removing a
> WindowAgg whose RPR WindowFuncs are all unused. Doing that precisely means
> restructuring remove_unused_subquery_outputs(), which runs for every
> subquery and not just RPR -- broad enough that I'm treating it as a
> longer-term item rather than part of this work. It doesn't bring the
> external call back -- the Var-only path stays -- so the revert to static is
> safe independently of it.
> 
> I'll fold the static revert into v48.

Thank for eplaining the history.

BTW, in v47-0002 patch,
there are some non ASCII characters ("§").

+	/*
+	 * Qualified column references in DEFINE are not supported.  This covers
+	 * both FROM-clause range variables (prohibited by §6.5) and pattern
+	 * variable qualified names (e.g. UP.price), which are valid per §4.16
+	 * but not yet implemented.
+	 */

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  351. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-01T02:29:21Z

    Hi Tatsuo,

> BTW, in v47-0002 patch,
> there are some non ASCII characters ("§").

Good catch -- that one's already handled on top of v47. nocfbot-0004
("Reclassify DEFINE qualifier check and reword diagnostic to 'expression'")
reworded that comment, replacing the section signs with a spelled-out
reference: "prohibited by §6.5" became "prohibited by ISO/IEC 19075-5 6.5",
and likewise for §4.16. The signs you're looking at are an artifact of the
base v47-0002 patch, before that incremental.

To be sure it wasn't lurking elsewhere, I re-swept the whole RPR footprint
-- the sources, the headers, README.rpr, and the regression tests -- and
there are no non-ASCII characters left in the current tree. The fix folds
into v48.

Thanks,
Henson

  352. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-01T02:47:03Z

    Hi Henson,

>> BTW, in v47-0002 patch,
>> there are some non ASCII characters ("§").
> 
> Good catch -- that one's already handled on top of v47. nocfbot-0004
> ("Reclassify DEFINE qualifier check and reword diagnostic to 'expression'")
> reworded that comment, replacing the section signs with a spelled-out
> reference: "prohibited by §6.5" became "prohibited by ISO/IEC 19075-5 6.5",
> and likewise for §4.16. The signs you're looking at are an artifact of the
> base v47-0002 patch, before that incremental.
> 
> To be sure it wasn't lurking elsewhere, I re-swept the whole RPR footprint
> -- the sources, the headers, README.rpr, and the regression tests -- and
> there are no non-ASCII characters left in the current tree. The fix folds
> into v48.

Oh, I see. Sorry for noise.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  353. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-01T05:35:24Z

    Hi Tatsuo, Jian,

While tidying RPR comments I found a small inconsistency in the varId
bounds.
The comment/README side I'm already fixing in the in-progress series;
whether
to also change the bounds is a separate follow-up.  As lead author that one
is
ultimately your call, Tatsuo, but I'd welcome Jian's and the list's input on
it first.

The current state, in src/include/optimizer/rpr.h:

  #define RPR_VARID_MAX   251
  #define RPR_VARID_BEGIN 252   /* control codes 252..255 */
  ... END 253, ALT 254, FIN 255

  RPRElemIsVar(e)  ==  ((e)->varId <= RPR_VARID_MAX)   /* 0..251 */

and the limit enforced in parse_rpr.c:

  if (list_length(*varNames) >= RPR_VARID_MAX)   /* reject the 252nd */
      ereport(ERROR, "too many pattern variables", "Maximum is 251");

So 251 variables are accepted as varId 0..250, leaving 251 a hole: never
assigned, yet the macro still classifies it as a variable -- one wider than
the comment's own "0 to RPR_VARID_MAX - 1".

RPRVarId is a uint8, kept small on purpose: varId is the likely per-row
match-history key, and since a match can run arbitrarily long the history
grows with it -- so one byte per row, not two, is what keeps that footprint
in check.

The catch of staying in uint8: the four control codes already fill 252..255,
so 251 is the only free slot for any future sentinel (anchor ^/$, exclusion
{- -}) short of widening to uint16.  So the hole is really the last reserve.

Three ways, by what the gap is spent on:

(1) Leave it -- just the doc alignment already underway: 251 stays a
documented
    reserve, macro unchanged.  No follow-up commit.  The one free slot is
then
    on hand for a single future control code, should one ever be needed.

(2) Fill it as a 252nd variable (0..251).  Compatible and doable anytime; a
few
    lines in parse_rpr.c / rpr.h plus the boundary test.  But it spends the
    last free slot, so a future control code would then force either a
    compatibility-breaking narrow of RPR_VARID_MAX or a widen to two bytes
    (doubling history).  Maximal variables now, the control question
deferred.

(3) Reserve 16 control codes now (4 used + 12 spare) at the 0xF0 boundary:
    vars 0..239, control 240..255, existing sentinels unmoved, macro becomes
    (varId & 0xF0) != 0xF0.  Buys 12-code headroom inside the byte, so
history
    stays 1 byte and (2)'s fork never arises.  Same edit shape as (2); costs
    only the nominal drop to 240 variables -- but it is a narrowing, so free
    only pre-release.

The asymmetry: (3) is the only one with a deadline -- a narrowing is
compatible
only before release, while (1)/(2) stay open forever.  So the question is
whether to spend this one free moment to lock in 1-byte control headroom
(3),
or stay minimal now (1)/(2) and take the narrow-or-widen later if it is ever
needed.  My own lean is toward (3): 240 variables is already far more than
any
real pattern will use, so the capacity we give up is nominal, while the
12-code
buffer closes the narrow-or-widen fork for good and keeps match history at
one
byte -- and it is the one choice that is free only now.  That said, I'd like
the decision to rest on everyone's input -- Jian's and the list's as much as
mine -- with you, Tatsuo, weighing it all and making the final call.

Either way, once the feature matures and the final control-code count is
known,
the space can be repacked gap-free -- so none of these is the last word.

Which would you prefer?

Thanks,
Henson

  354. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-02T02:50:39Z

    Hi Henson, Jian,

> Hi Tatsuo, Jian,
> 
> While tidying RPR comments I found a small inconsistency in the varId
> bounds.
> The comment/README side I'm already fixing in the in-progress series;
> whether
> to also change the bounds is a separate follow-up.  As lead author that one
> is
> ultimately your call, Tatsuo, but I'd welcome Jian's and the list's input on
> it first.
> 
> The current state, in src/include/optimizer/rpr.h:
> 
>   #define RPR_VARID_MAX   251
>   #define RPR_VARID_BEGIN 252   /* control codes 252..255 */
>   ... END 253, ALT 254, FIN 255
> 
>   RPRElemIsVar(e)  ==  ((e)->varId <= RPR_VARID_MAX)   /* 0..251 */
> 
> and the limit enforced in parse_rpr.c:
> 
>   if (list_length(*varNames) >= RPR_VARID_MAX)   /* reject the 252nd */
>       ereport(ERROR, "too many pattern variables", "Maximum is 251");
> 
> So 251 variables are accepted as varId 0..250, leaving 251 a hole: never
> assigned, yet the macro still classifies it as a variable -- one wider than
> the comment's own "0 to RPR_VARID_MAX - 1".
> 
> RPRVarId is a uint8, kept small on purpose: varId is the likely per-row
> match-history key, and since a match can run arbitrarily long the history
> grows with it -- so one byte per row, not two, is what keeps that footprint
> in check.
> 
> The catch of staying in uint8: the four control codes already fill 252..255,
> so 251 is the only free slot for any future sentinel (anchor ^/$, exclusion
> {- -}) short of widening to uint16.  So the hole is really the last reserve.
> 
> Three ways, by what the gap is spent on:
> 
> (1) Leave it -- just the doc alignment already underway: 251 stays a
> documented
>     reserve, macro unchanged.  No follow-up commit.  The one free slot is
> then
>     on hand for a single future control code, should one ever be needed.
> 
> (2) Fill it as a 252nd variable (0..251).  Compatible and doable anytime; a
> few
>     lines in parse_rpr.c / rpr.h plus the boundary test.  But it spends the
>     last free slot, so a future control code would then force either a
>     compatibility-breaking narrow of RPR_VARID_MAX or a widen to two bytes
>     (doubling history).  Maximal variables now, the control question
> deferred.
> 
> (3) Reserve 16 control codes now (4 used + 12 spare) at the 0xF0 boundary:
>     vars 0..239, control 240..255, existing sentinels unmoved, macro becomes
>     (varId & 0xF0) != 0xF0.  Buys 12-code headroom inside the byte, so
> history
>     stays 1 byte and (2)'s fork never arises.  Same edit shape as (2); costs
>     only the nominal drop to 240 variables -- but it is a narrowing, so free
>     only pre-release.
> 
> The asymmetry: (3) is the only one with a deadline -- a narrowing is
> compatible
> only before release, while (1)/(2) stay open forever.  So the question is
> whether to spend this one free moment to lock in 1-byte control headroom
> (3),
> or stay minimal now (1)/(2) and take the narrow-or-widen later if it is ever
> needed.  My own lean is toward (3): 240 variables is already far more than
> any
> real pattern will use, so the capacity we give up is nominal, while the
> 12-code
> buffer closes the narrow-or-widen fork for good and keeps match history at
> one
> byte -- and it is the one choice that is free only now.  That said, I'd like
> the decision to rest on everyone's input -- Jian's and the list's as much as
> mine -- with you, Tatsuo, weighing it all and making the final call.
> 
> Either way, once the feature matures and the final control-code count is
> known,
> the space can be repacked gap-free -- so none of these is the last word.
> 
> Which would you prefer?

I'd prefer (3). Yes, I agree that 240 pattern variables is enough.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  355. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-02T03:03:05Z

    Hi Henson, Jian,

I'd like to suggest making a slight change to v48 patches
organization. Currently README.rpr is included in 0005 executor
patch. Since README.rpr is a documentation, I think it will be more
natural to move it into 0006 docs patch. This also will make it easier
to judge the size of the patches by category: codes, docs and tests if
README.rpr is correctly categorized.

Thoughts?
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  356. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-02T03:05:41Z

    2026년 6월 2일 (화) 12:03, Tatsuo Ishii <ishii@postgresql.org>님이 작성:

> Hi Henson, Jian,
>
> I'd like to suggest making a slight change to v48 patches
> organization. Currently README.rpr is included in 0005 executor
> patch. Since README.rpr is a documentation, I think it will be more
> natural to move it into 0006 docs patch. This also will make it easier
> to judge the size of the patches by category: codes, docs and tests if
> README.rpr is correctly categorized.
>

+1


> Thoughts?
> --
> Tatsuo Ishii
> SRA OSS K.K.
> English: http://www.sraoss.co.jp/index_en/
> Japanese:http://www.sraoss.co.jp
>

  357. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-02T05:20:09Z

    Hi Henson,

> Hi all,
> 
> Resolved since the last post:
> 
>   D1. Single-row frame conformance, Subclause 6.10.2 -- Tatsuo's call [6]
>       was to reject. Both ROWS BETWEEN CURRENT ROW AND CURRENT ROW and
>       AND 0 FOLLOWING are now rejected (nocfbot-0025), which in turn
>       unblocks the held ExecRPRProcessRow change (nocfbot-0026).
> 
> Open decisions (repeated with context at the end):
> 
>   D2. The RPRContext consolidation -- Tatsuo's call as co-author;
>       non-blocking either way [4].
>   D3. The AST "absorption" rename -- Tatsuo's call [2].
> 
>   [1] Jian's review, round 1 (2026-05-26):
> 
> https://postgr.es/m/CACJufxH-DZePhbdJM=8nNYceQiSbbXXLTw54iLhxiynQ+4hbBA@mail.gmail.com
>   [2] my round-1 reply -- AST "absorption" rename deferred to Tatsuo (D3,
> 2026-05-27):
> 
> https://postgr.es/m/CAAAe_zDephfiDA_A3FN0hCymJRogEr=Rt3QoCTf4qMYDLk+xNA@mail.gmail.com
>   [3] Jian's review, round 2 (2026-05-28):
> 
> https://postgr.es/m/CACJufxGX17thWuEOq1tM5xbRHz2HXm1asooZC3GV25MYGmYqLQ@mail.gmail.com
>   [4] my round-2 reply -- RPRContext deferred to Tatsuo (D2, 2026-05-29):
> 
> https://postgr.es/m/CAAAe_zAH0MvP0TBmW3PTLeHjEpiyBz0473zJRM9pwLpseefMNw@mail.gmail.com
>   [5] single-row frame conformance question (D1, 2026-05-29):
> 
> https://postgr.es/m/CAAAe_zCbSU=dd-4qTL2QaBQwQ-cf51N_851a9Y5rOoz0wj0aXw@mail.gmail.com
>   [6] Tatsuo's reply -- reject single-row frames (D1 resolved, 2026-05-30):
> 
> https://postgr.es/m/20260530.114737.1416684464524168377.ishii@postgresql.org
>   [7] Jian's review, round 3 (2026-05-30):
> 
> https://postgr.es/m/CACJufxEsaU8GQ4yeXTWhAO8VjbrZTh5CpvUqz=4a3T0Cwz44pA@mail.gmail.com
> 
> Attached: the v47 feature series (v47-0001..0009) rebased onto current
> master, plus the incremental patch series carried on top of it.
> 
> Base:
> 
>   9a41b34a287  2026-05-26  doc:  add comma to UPDATE docs, for consistency
> 
> Unchanged -- rebase only (already posted; only rebased, no content change).
> Titles for reference:
> 
>   nocfbot-0001  Add DEFINE non-volatile baseline to rpr_integration B9
>   nocfbot-0002  Unify RPR DEFINE walkers and reject volatile callees

While looking into 0002, I noticed some minor ereport calling style
issues.

Recently we start to use the style:

		ereport(ERROR,
				errcode(ERRCODE_FEATURE_NOT_SUPPORTED),

Rather than (old style):

		ereport(ERROR,
				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),

Grepping patches shows following results that hire the old style.

$ grep -n '(errcode' *|grep '+'
nocfbot-0002-Unify-RPR-DEFINE-walkers-and-reject-volatile-call.txt:1243:+				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
nocfbot-0002-Unify-RPR-DEFINE-walkers-and-reject-volatile-call.txt:1248:+				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
nocfbot-0002-Unify-RPR-DEFINE-walkers-and-reject-volatile-call.txt:1421:+								(errcode(ERRCODE_SYNTAX_ERROR),
nocfbot-0002-Unify-RPR-DEFINE-walkers-and-reject-volatile-call.txt:1428:+								(errcode(ERRCODE_SYNTAX_ERROR),
nocfbot-0002-Unify-RPR-DEFINE-walkers-and-reject-volatile-call.txt:1451:+							(errcode(ERRCODE_SYNTAX_ERROR),
nocfbot-0002-Unify-RPR-DEFINE-walkers-and-reject-volatile-call.txt:1457:+							(errcode(ERRCODE_SYNTAX_ERROR),
nocfbot-0002-Unify-RPR-DEFINE-walkers-and-reject-volatile-call.txt:1463:+							(errcode(ERRCODE_SYNTAX_ERROR),
nocfbot-0002-Unify-RPR-DEFINE-walkers-and-reject-volatile-call.txt:1471:+						(errcode(ERRCODE_SYNTAX_ERROR),
nocfbot-0002-Unify-RPR-DEFINE-walkers-and-reject-volatile-call.txt:1494:+							(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
nocfbot-0002-Unify-RPR-DEFINE-walkers-and-reject-volatile-call.txt:1504:+							(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
nocfbot-0025-Reject-single-row-window-frame-in-row-pattern-rec.txt:55:+						(errcode(ERRCODE_WINDOWING_ERROR),

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  358. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-02T05:23:56Z

    Oops.

> While looking into 0002, I noticed some minor ereport calling style
> issues.
> 
> Recently we start to use the style:
> 
> 		ereport(ERROR,
> 				errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
> 
> Rather than (old style):
> 
> 		ereport(ERROR,
> 				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
> 
> Grepping patches shows following results that hire the old style.
> 
> $ grep -n '(errcode' *|grep '+'
> nocfbot-0002-Unify-RPR-DEFINE-walkers-and-reject-volatile-call.txt:1243:+				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
> nocfbot-0002-Unify-RPR-DEFINE-walkers-and-reject-volatile-call.txt:1248:+				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
> nocfbot-0002-Unify-RPR-DEFINE-walkers-and-reject-volatile-call.txt:1421:+								(errcode(ERRCODE_SYNTAX_ERROR),
> nocfbot-0002-Unify-RPR-DEFINE-walkers-and-reject-volatile-call.txt:1428:+								(errcode(ERRCODE_SYNTAX_ERROR),
> nocfbot-0002-Unify-RPR-DEFINE-walkers-and-reject-volatile-call.txt:1451:+							(errcode(ERRCODE_SYNTAX_ERROR),
> nocfbot-0002-Unify-RPR-DEFINE-walkers-and-reject-volatile-call.txt:1457:+							(errcode(ERRCODE_SYNTAX_ERROR),
> nocfbot-0002-Unify-RPR-DEFINE-walkers-and-reject-volatile-call.txt:1463:+							(errcode(ERRCODE_SYNTAX_ERROR),
> nocfbot-0002-Unify-RPR-DEFINE-walkers-and-reject-volatile-call.txt:1471:+						(errcode(ERRCODE_SYNTAX_ERROR),
> nocfbot-0002-Unify-RPR-DEFINE-walkers-and-reject-volatile-call.txt:1494:+							(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
> nocfbot-0002-Unify-RPR-DEFINE-walkers-and-reject-volatile-call.txt:1504:+							(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),

These have been fixed in 0007. Sorry for noise.

Maybe this is the only one worth to look into?

> nocfbot-0025-Reject-single-row-window-frame-in-row-pattern-rec.txt:55:+						(errcode(ERRCODE_WINDOWING_ERROR),

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  359. Re: Row pattern recognition

    jian he <jian.universality@gmail.com> — 2026-06-02T05:46:22Z

    On Mon, Jun 1, 2026 at 1:35 PM Henson Choi <assam258@gmail.com> wrote:
>
> Hi Tatsuo, Jian,
>
> While tidying RPR comments I found a small inconsistency in the varId bounds.
> The comment/README side I'm already fixing in the in-progress series; whether
> to also change the bounds is a separate follow-up.  As lead author that one is
> ultimately your call, Tatsuo, but I'd welcome Jian's and the list's input on
> it first.
>
> The current state, in src/include/optimizer/rpr.h:
>
>   #define RPR_VARID_MAX   251
>   #define RPR_VARID_BEGIN 252   /* control codes 252..255 */
>   ... END 253, ALT 254, FIN 255
>
>   RPRElemIsVar(e)  ==  ((e)->varId <= RPR_VARID_MAX)   /* 0..251 */
>
> and the limit enforced in parse_rpr.c:
>
>   if (list_length(*varNames) >= RPR_VARID_MAX)   /* reject the 252nd */
>       ereport(ERROR, "too many pattern variables", "Maximum is 251");
>
> So 251 variables are accepted as varId 0..250, leaving 251 a hole: never
> assigned, yet the macro still classifies it as a variable -- one wider than
> the comment's own "0 to RPR_VARID_MAX - 1".
>
> RPRVarId is a uint8, kept small on purpose: varId is the likely per-row
> match-history key, and since a match can run arbitrarily long the history
> grows with it -- so one byte per row, not two, is what keeps that footprint
> in check.
>
> The catch of staying in uint8: the four control codes already fill 252..255,
> so 251 is the only free slot for any future sentinel (anchor ^/$, exclusion
> {- -}) short of widening to uint16.  So the hole is really the last reserve.
>
> Three ways, by what the gap is spent on:
>
> (1) Leave it -- just the doc alignment already underway: 251 stays a documented
>     reserve, macro unchanged.  No follow-up commit.  The one free slot is then
>     on hand for a single future control code, should one ever be needed.
>
> (2) Fill it as a 252nd variable (0..251).  Compatible and doable anytime; a few
>     lines in parse_rpr.c / rpr.h plus the boundary test.  But it spends the
>     last free slot, so a future control code would then force either a
>     compatibility-breaking narrow of RPR_VARID_MAX or a widen to two bytes
>     (doubling history).  Maximal variables now, the control question deferred.
>
> (3) Reserve 16 control codes now (4 used + 12 spare) at the 0xF0 boundary:
>     vars 0..239, control 240..255, existing sentinels unmoved, macro becomes
>     (varId & 0xF0) != 0xF0.  Buys 12-code headroom inside the byte, so history
>     stays 1 byte and (2)'s fork never arises.  Same edit shape as (2); costs
>     only the nominal drop to 240 variables -- but it is a narrowing, so free
>     only pre-release.
>
> Which would you prefer?
>

3.

240 variables is enough, as each variable supports multiple complex AND/OR
conditions. Additionally, since PostgreSQL regular expressions use 14 special
characters, reserving the remaining ones in advance is a future-proof approach.

----------------------------------------------------
src/backend/executor/README.rpr

XII-4. Memory Pool Management
  Choice: Custom free list

  Rationale:
  - NFA states are created and destroyed in large numbers per row
  - Avoids palloc/pfree overhead
  - State size is variable (counts[] array), but within a single query
    maxDepth is fixed, so all states have the same size

It would be better simply to mention that:
RPRNFAState and RPRNFAContext are allocated in a partition-lifespan
memory context; they will be destroyed in release_partition.
--------------------------
in ExecRPRFreeContext:
{
    if (ctx->states != NULL)
        nfa_state_free_list(winstate, ctx->states);
    if (ctx->matchedState != NULL)
        nfa_state_free(winstate, ctx->matchedState);
}

If ctx->matchedState points to one of the states already in ctx->states, will
nfa_state_free() be called on the same RPRNFAState twice? Is this double-free
permitted, or do we have a mechanism in place to guard against it?
--------------------------
In ExecRPRProcessRow

            if (currentPos == ctxFrameEnd)
            {
                /* Frame boundary reached: force mismatch */
                nfa_match(winstate, ctx, NULL);
                continue;
            }

If I comment out the CONTINUE, the entire regression still succeeds.



--
jian
https://www.enterprisedb.com/

  360. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-02T10:19:12Z

    Hi Henson,

In execRPR.c:

static void
nfa_reevaluate_dependent_vars(WindowAggState *winstate, RPRNFAContext *ctx,
							  int64 currentPos)

This function does not have a forward declaration. Moreover, it is put
in the "API exposed to nodeWindowAgg.c" section. If there's no
particular reason for this, I suggest to add a forward declaration for
it and move it before "API exposed to nodeWindowAgg.c" section.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  361. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-02T14:11:05Z

    Hi Tatsuo,

Catching up on today's notes -- all straightforward, so I'm folding them
into one reply.

> I'd prefer (3). Yes, I agree that 240 pattern variables is enough.

Then (3) it is -- and with Jian on the same choice, that closes it. I'll
go ahead with it. Since it's a narrowing it's only free pre-release, so
it goes in the v48 core rather than a later follow-up.

> static void
> nfa_reevaluate_dependent_vars(WindowAggState *winstate, RPRNFAContext
*ctx,
>                               int64 currentPos)
>
> This function does not have a forward declaration. Moreover, it is put
> in the "API exposed to nodeWindowAgg.c" section. [...] I suggest to add
> a forward declaration for it and move it before "API exposed to
> nodeWindowAgg.c" section.

Agreed -- that was just an oversight. I'll add the forward declaration
and move it out of the API-exposed section.

> nocfbot-0025-...:55:+   (errcode(ERRCODE_WINDOWING_ERROR),
> Maybe this is the only one worth to look into?

Right -- I'll update 0025 in the next posting.

Thanks,
Henson

  362. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-02T22:14:21Z

    Subject: Re: Row pattern recognition

Hi Jian,

> 3.
> 240 variables is enough, as each variable supports multiple complex
> AND/OR conditions. [...] reserving the remaining ones in advance is a
> future-proof approach.

Thanks -- that's (3), and Tatsuo landed on the same choice, so it's
settled. I'll make the change for v48.

> XII-4. Memory Pool Management
>   Choice: Custom free list [...]
> It would be better simply to mention that:
> RPRNFAState and RPRNFAContext are allocated in a partition-lifespan
> memory context; they will be destroyed in release_partition.

Agreed -- that's clearer than the rationale list. I'll replace XII-4 with
that wording.

> If ctx->matchedState points to one of the states already in ctx->states,
> will nfa_state_free() be called on the same RPRNFAState twice? Is this
> double-free permitted, or do we have a mechanism in place to guard
> against it?

No -- they're disjoint by construction, so the two frees never touch the
same state. Step by step:

  - nfa_advance() detaches ctx->states up front (sets it to NULL) and
    rebuilds the list from scratch.
  - Each old state is pulled off and advanced; the one that reaches FIN is
    moved into matchedState and never re-added to ctx->states.
  - So at any moment a state is either on ctx->states or it is
    matchedState, never both.
  - ExecRPRFreeContext frees the list and frees matchedState -- disjoint
    sets, no overlap, so no guard is needed.
  - (When a new FIN replaces matchedState, the previous one is freed right
    there.)

> if (currentPos == ctxFrameEnd) {
>     nfa_match(winstate, ctx, NULL);
>     continue;
> }
> If I comment out the CONTINUE, the entire regression still succeeds.

It isn't dead -- it's the N-FOLLOWING boundary handler. nfa_match(ctx,
NULL) forces a mismatch that finalizes the context on its matchedState,
and the continue skips the rest since it's done for this row.

Without the continue, that finalized context is matched a second time in
the same row -- a context-matched-twice error. It only looks harmless
because the forced mismatch already emptied ctx->states, so the second
nfa_match iterates nothing (and it would also re-run
nfa_reevaluate_dependent_vars(), overwriting the shared nfaVarMatched the
other contexts read). So I'd keep it: a finalized context must not be
matched twice in the same row.

Thanks,
Henson

  363. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-02T23:50:28Z

    Hi all,

While going over the row pattern grammar I want to put on record why the
empty pattern -- PATTERN (()) and the like -- is left unsupported, and why
I think that is the right call for this series rather than an oversight.

Today it is simply a syntax error.  row_pattern_primary is either a
variable (ColId) or a parenthesized group '(' row_pattern ')', and
row_pattern has no empty production, so '()' never parses.  The standard's
pattern syntax does allow an empty row pattern -- it matches the empty
sequence, i.e. it produces an empty match -- so the question is whether we
should grow the grammar, plus an NFA "empty" element, to accept it.

My claim is that we do not need to: every way an empty pattern can appear
reduces to something we already handle, so a dedicated empty element in
the executor would be dead weight.  There are two cases.


1. The empty pattern is the whole pattern: PATTERN (())

This pattern has zero pattern variables.  But DEFINE is mandatory (per
ISO/IEC 19075-5, Table 18), an empty DEFINE list is rejected, and every
DEFINE variable must appear in PATTERN -- otherwise we already error with
"DEFINE variable \"%s\" is not used in PATTERN".  A pattern with no
variables cannot satisfy any of that: there is nothing for DEFINE to
define, yet DEFINE can be neither omitted nor left empty.  So an all-empty
pattern is rejected by the existing rules, with no new check needed.

(It is degenerate in any case.  An empty match at every row maps to "no
reduced frame" -- row_is_in_reduced_frame() returns -1 for RF_EMPTY_MATCH
exactly as it does for RF_UNMATCHED -- so every row would simply be
unmatched.)


2. The empty pattern is embedded: A () B

Here '()' consumes no rows; it is the identity element under
concatenation, so A () B is equivalent to A B.  This stays entirely on the
parser/optimizer side: the empty-pattern production would carry an empty
AST node, and an identity fold in the SEQ simplification (next to the
prefix/suffix and consecutive merges it already does) drops the '()'
before the executor ever sees it.  The runtime is untouched -- which is
the whole point.  And if the fold removes everything, the pattern has
collapsed to case 1 and is rejected there.

The AST rewrites it would add:

  Concatenation (identity element):
    A () B          ->  A B
    () A            ->  A
    A ()            ->  A
    A () () B       ->  A B

  Group unwrap (single non-empty child):
    (A ())          ->  (A)  ->  A
    (() A)          ->  (A)  ->  A

  Quantified empty (empty repeated is still empty):
    ()*  ()+  ()?  (){3}    ->  (removed)
    A ()* B         ->  A B

  Collapses to case 1, then rejected:
    (())    (() ())    ((()))    ->  ()  -> rejected as case 1

The one form that is not pure deletion is an empty alternative.  Here the
empty branch is optionality, not nothing, but it is exactly the '?'
quantifier applied to the rest -- X | () is (X)?:

    X | ()          ->  (X)?          (and  () | X  ->  (X)??, reluctant)
      A | ()        ->  A?            (single var: group is redundant)
      A{2} | ()     ->  (A{2})?       (the group is required here -- A{2}?
                                       would parse as a reluctant A{2}, not
                                       an optional one)
      A B | ()      ->  (A B)?         (X may be a sequence)
      A | B | ()    ->  (A | B)?       (... or an alternation)
    (A | ()) B      ->  A? B

The branch order maps to greediness ('X | ()' greedy, '() | X' reluctant).
Either way it rewrites to a group plus '?', both of which we already
support, so it still needs no new machinery.

The rewrites also compose across nesting.  A nested empty alternation
applies the rule at each level, producing a nested nullable quantifier:

    (A | ()) | ()
        ->  A? | ()      (inner   A | ()  ->  A?)
        ->  (A?)?        (outer   X | ()  ->  (X)?,  X = A?)

and (A?)? = A?, (A+)? = A*, (A*)? = A* are all ordinary nullable
constructs, never a new shape:

    A? | ()   ->  (A?)?    (= A?)
    A+ | ()   ->  (A+)?    (= A*)
    A* | ()   ->  (A*)?    (= A*)

Whether the optimizer collapses these to a single quantifier or leaves the
group as-is doesn't matter here -- both are nullable groups the runtime
already handles, so still no executor-level element.  (The optimizer
leaves these unfolded: a non-exact outer over a nullable child falls
outside tryMultiplyQuantifiers' conservative guard, which skips the whole
class that can have gaps (e.g. (A{2}){2,3} = {4,6}).  That is an
over-rejection, not a correctness defect -- folding the nullable cases
would be a safe optional extension; see the aside below.)

(Duplicate empty branches would be merged first by the alternation dedup,
so A | () | () reduces through A | () to A? just the same.)


Putting it together: across the board an empty pattern is either

  (a) rejected by the existing DEFINE rules (case 1),
  (b) the concatenation identity, removable by the AST simplification
      (case 2), or
  (c) an existing nullable construct ('?', the empty alternative),

and none of these would ever need an executor-level empty element.

Given that, and that the feature has little practical use while the
current series is already sizable, I am leaving the empty pattern out of
scope: it stays a syntax error.  Concretely, accepting it would take a
grammar production, an empty AST node, the concatenation-identity deletion
in the SEQ pass, and (optionally) an extension of the quantifier
composition to fold the nullable cases -- all new work, none of it in this
series.  We can revisit it as that small follow-up if anyone actually
wants it.

Thanks,
Henson


------------------------------------------------------------------------
Aside (separable from the post above): quantifier-composition gaps,
unrelated to the empty pattern
------------------------------------------------------------------------

Independent of the empty pattern, the AST-level quantifier-composition
pass (tryMultiplyQuantifiers, rpr.c) over-rejects: it collapses a nested
quantifier (X{p,q}){m,n} into a single X{...} only when the outer is exact
(m == n), the child is {1,1}, or both are unbounded.  Those are the
always-safe sufficient conditions.  But the full foldable class is larger:
(X{p,q}){m,n} is foldable to X{m*p, n*q} whenever the reachable repetition
counts -- the union over k in [m,n] of [k*p, k*q] -- are contiguous (no
gaps).  That holds for every nullable child (p == 0) and, more generally,
whenever the child span q-p is wide enough to bridge the jump between k and
k+1 iterations.  The guard thus rejects a strict superset of the genuinely
unsafe (gap-producing) patterns: a class of provably-safe folds is left
undone, and those patterns reach the executor as nested groups -- handled
correctly, just not normalized.  This is an optimization over-rejection,
not a correctness defect: the output is identical either way.

The two rules side by side (outer {m,n} over child {p,q}, INF = unbounded):

  current guard -- folds iff:
      m = n                                       (outer exact)
    OR (p = 1 AND q = 1)                          (child {1,1})
    OR (q = INF AND n = INF AND NOT(m = 0 AND p >= 2))   (both unbounded)

  true safe set -- foldable iff the reachable counts are contiguous:
      m = n                                       (single interval)
    OR p = 0                                      (nullable child)
    OR ( p <= max(m,1)*(q - p) + 1                (child span bridges the
         AND (m >= 1 OR p <= 1) )                  k -> k+1 jump; child
{1,1}
                                                   is the p=q=1 instance)

  over-rejected = safe AND NOT current:
    the non-exact-outer (m != n) folds over a gap-free child -- p = 0
    (nullable) or a span-bridging child.  e.g. (A?)?, (A*)?, (A+)?,
    (A?){2,3}, (A+){0,2}, (A{1,3}){2,4} (all in the table below).
    The boundaries (A{2}){2,3} and (A{2,})* are NOT in the safe set
    (p = 2 fails p <= max(m,1)*(q-p)+1), so both rules reject them --
    correctly; widening the guard must keep doing so.

Verified on the branch -- EXPLAIN's "Pattern:" line shows the optimized
form (a nested group means it was not folded; a flat quantifier means it
was):

  folds today (correct):
    (A){2,3}        -> a{2,3}        (child {1,1})
    (A{2,3}){2}     -> a{4,6}        (outer exact)
    (A*)*           -> a*            (both unbounded)

  gap-free but NOT folded (stays nested; "want" is the safe normal form):
    (A?)?           -> (a?)?         want a?
    (A*)?           -> (a*)?         want a*
    (A+)?           -> (a+)?         want a*
    (A?){2,3}       -> (a?){2,3}     want a{0,3}
    (A+){0,2}       -> (a+){0,2}     want a*
    (A{1,3}){2,4}   -> (a{1,3}){2,4} want a{2,12}

  correctly NOT foldable (real gap -- guard is right here):
    (A{2}){2,3}     -> (a{2}){2,3}   reaches {4,6}, not 4..6

The first three "not folded" rows are the nullable forms the empty pattern
would lean on; the next three show the gap is broader (a range outer over
a non-{1,1} child can still be gap-free).  The win is only normalization /
a smaller NFA, so this is low priority.  And this exact pass has a history
of subtle gap bugs (the (A{2,})* over-flatten and the INF-sum merge, both
recently fixed), so widening the guard toward the safe set above should be
its own patch, with boundary tests covering the foldable cases and the two
gaps -- (A{2}){2,3} and (A{2,})* -- that must stay un-folded.

  364. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-04T04:21:08Z

    Hi Henson,

Just a trivial comment to 1.

> While going over the row pattern grammar I want to put on record why the
> empty pattern -- PATTERN (()) and the like -- is left unsupported, and why
> I think that is the right call for this series rather than an oversight.
> 
> Today it is simply a syntax error.  row_pattern_primary is either a
> variable (ColId) or a parenthesized group '(' row_pattern ')', and
> row_pattern has no empty production, so '()' never parses.  The standard's
> pattern syntax does allow an empty row pattern -- it matches the empty
> sequence, i.e. it produces an empty match -- so the question is whether we
> should grow the grammar, plus an NFA "empty" element, to accept it.
> 
> My claim is that we do not need to: every way an empty pattern can appear
> reduces to something we already handle, so a dedicated empty element in
> the executor would be dead weight.  There are two cases.
> 
> 
> 1. The empty pattern is the whole pattern: PATTERN (())
> 
> This pattern has zero pattern variables.  But DEFINE is mandatory (per
> ISO/IEC 19075-5, Table 18), an empty DEFINE list is rejected, and every
> DEFINE variable must appear in PATTERN -- otherwise we already error with
> "DEFINE variable \"%s\" is not used in PATTERN".  A pattern with no
> variables cannot satisfy any of that: there is nothing for DEFINE to
> define, yet DEFINE can be neither omitted nor left empty.  So an all-empty
> pattern is rejected by the existing rules, with no new check needed.

I think current behavior is correct from the standard's point of
view. As you explained, there's no way to accept "PATTERN (())" or
"PATTERN ()" according to the R020 syntax rule.

Note, however, "PATTERN ()" is possible in R010 because it allows to
omit the DEFINE clause itself.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  365. Re: Row pattern recognition

    jian he <jian.universality@gmail.com> — 2026-06-04T09:20:07Z

    Hi.

Please check the attached regression test refactoring and gram.y changes

------------------atatched patch commit
message-----------------------------------------------
v47 rpr reformat gram.y and improve regression tests

Reformat the first three alternatives of row_pattern_quantifier_opt
in gram.y (empty, '*', '+') from inline single-line actions to
multi-line blocks, making them consistent with the surrounding Op
and '{...}' alternatives.

Update regression tests to follow the convention established in
commit ecb2508aaf [1]: do not repeat exact error messages as comments
in regress test files, since such comments silently go stale when
error wording changes.  Replace the removed "Expected: ERROR: ..."
comments with short descriptive comments that state what the test
is checking rather than what output it expects.

Also remove unnecessary newline/comments in src/test/regress/sql/rpr_base.sql.

Add test coverage for:
1. invalid token combinations after a quantifier (A+ !, A+ ?+,
  A* ?+, A? ??), exercising the split-token error paths in
  row_pattern_quantifier_opt
2. set-returning function (generate_series) in a DEFINE clause

[1]: https://git.postgresql.org/cgit/postgresql.git/commit/?id=ecb2508aaf9b978871734ea2fdf701ab7d593d0a
----------------------------end of commit
message-------------------------------------------------

It's based on latest commit in https://github.com/assam258-5892/postgres.git
It will have a small conflict (around 100 lines difference) with v47.

In src/test/regress/sql/rpr_base.sql, wording such as ``Jacob's
Patterns`` should be removed?

```
--   Serialization/Deserialization Tests (objects kept for pg_upgrade/pg_dump)
```
I am not sure what this refers to.

In gram.y:
errmsg("quantifier bound must be between 0 and %d", INT_MAX - 1),
errmsg("quantifier bound must be between 1 and %d", INT_MAX - 1),

Will these cause consistency issues?



--
jian
https://www.enterprisedb.com/

  366. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-04T12:15:40Z

    > Please check the attached regression test refactoring and gram.y changes

Thanks for the patch -- the substance all looks right: the gram.y reformat
is a fair consistency fix, the ecb2508aaf convention (no hardcoded error
text in regress comments) is one we should follow, and the added coverage
(split-token quantifier errors, SRF in DEFINE) fills real gaps.

I'll work these in together with the patches I'm lining up right before the
v48 merge, rather than mid-stream -- the large "Expected: ERROR" comment
cleanup would otherwise churn against the other test edits still in flight.
They'll all land in that v48 round.

Answers to your three questions:

> In src/test/regress/sql/rpr_base.sql, wording such as ``Jacob's
> Patterns`` should be removed?

Those came in from Jacob's branch, but agreed they're better changed:
"Jacob's Patterns" / "from jacob branch" are dev-time provenance notes, not
something to keep in committed tests. I'll give the section a descriptive
header (the tests themselves stay) and do it together with the v48 round
above.

> --   Serialization/Deserialization Tests (objects kept for
pg_upgrade/pg_dump)
> I am not sure what this refers to.

Those are views (with RPR in their definition) left undropped on purpose,
so the pg_dump/pg_upgrade tests pick them up: dumping and restoring the
regression DB deparses the RPR window clause and re-parses it, exercising
the serialization round trip. rpr_explain.sql keeps one for the same
reason. I'll reword the comment to say that.

> errmsg("quantifier bound must be between 0 and %d", INT_MAX - 1),
> errmsg("quantifier bound must be between 1 and %d", INT_MAX - 1),
> Will these cause consistency issues?

No, it's intentional. INT_MAX is the unbounded sentinel (RPR_QUANTITY_INF),
so an explicit bound is capped at INT_MAX - 1 to avoid colliding with it;
the "0 vs 1" split just matches each form ({n}/{,m} need >= 1, {n,} allows
0).

Regards,
Henson

  367. Re: Row pattern recognition

    Andres Freund <andres@anarazel.de> — 2026-06-04T20:38:36Z

    Hi,

cfbot is currently failing, with crashes inside JIT, on this patch. However,
the patch cfbot is submitting is rather old, because all of the versions
posted since have been marked nocfbot.

Greetings,

Andres Freund

  368. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-04T22:49:27Z

    > cfbot is currently failing, with crashes inside JIT, on this patch.

Thanks for letting know us. If my memory serves, the old cfbot did not
fail. So maybe the failure is due to something changed recently on the
master branch regarding JIT but the patch has not followed it yet?

Henson,
Do yo have any idea?

> However,
> the patch cfbot is submitting is rather old, because all of the versions
> posted since have been marked nocfbot.

Yes, it's intended. They are all incremental diffes against old (v47)
patches to save the patch size.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  369. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-04T23:24:05Z

    > Henson,
> Do you have any idea?

Let me be honest about this one.

I think the root issue is that none of us working on the patch has
enough confidence -- enough real control -- over the tuple slot and JIT
side of it. I wrote the code, it ran fine for me, and I assumed that
meant it was fine; but I have never felt that I fully understand and
control that area. So I cannot point to a specific cause with confidence
yet, and I would rather say that plainly than guess.

One practical note: there have been no tuple slot or JIT changes in the
patch since v47, and none are planned. So that part has been stable, and
reviewing the tuple slot / JIT code against v47 is perfectly viable --
there is no need to wait for a newer revision to look at that area.

What I think would genuinely harden the patch is for someone who knows
tuple slots and JIT well to review -- or rewrite -- that part. I would
very much welcome that, and I am happy to provide the design direction
behind the current code, or any other information a reviewer would need.

To be fair, the same caveat applies to the existing planner code the
patch builds on. I feel I have a reasonable grip on the rest of the
patch, but not on those two areas.

I will keep digging on my side (trying to reproduce the JIT crash with
JIT forced on against the current versions), and report back with
anything concrete I find.

Regards,
Henson

  370. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-09T04:31:57Z

    Hi hackers,

This is the v48 fold previewed in my 2026-05-30 post: the incremental
series carried on top of v47 has been folded back, Jian He's round-3
review [1] is applied (my point-by-point reply [2]), and an in-house
static-analysis pass plus the cosmetic clean-up are now included.

Resolved since the last post:

  D3. AST "absorption" rename -- Tatsuo's call.  Renamed to "merging"
      (nocfbot-0031).

  0025 ereport style (Tatsuo, 2026-06-02) [3]: the single-row frame
  rejection used the old (errcode(...)) wrapping; updated in place to the
  new errcode/errmsg style.  (The remaining RPR ereports are tidied up
  tree-wide in nocfbot-0045.)

Deferred (unchanged):

  D2. RPRContext consolidation (Jian's round-2 #1) -- folded into the
      future R010 infrastructure, where a single context can back both the
      R020 window implementation and R010.  We agreed to defer it at the v47
      stage (Tatsuo, 2026-05-31); it is not part of v48.

Attached: the v47 feature series (v47-0001..0009) rebased onto current
master, plus the full incremental series (nocfbot-0001..0068) carried
on top of it.

Base:

  89eafad297a  2026-06-06  Fix tuple deforming with virtual generated
columns

Already posted (nocfbot-0001..0026): 0001..0016 went out in earlier
rounds -- 0001..0011 [4], 0012..0015 [5], and 0016 (sent at the time as
0015) [6]; the 2026-05-30 post [7] renumbered them to run contiguously and
added 0017..0026.  All are rebase only and unchanged, except nocfbot-0025,
whose single-row-frame ereports were updated to the new errcode/errmsg
style (per Tatsuo; see above).  Titles for reference:

  nocfbot-0001  Add DEFINE non-volatile baseline to rpr_integration B9
  nocfbot-0002  Unify RPR DEFINE walkers and reject volatile callees
  nocfbot-0003  Cover RPR empty-match path with EXPLAIN tests; fix stale
                XXX comments
  nocfbot-0004  Reclassify DEFINE qualifier check and reword diagnostic to
                "expression"
  nocfbot-0005  Sync stale comments on DEFINE/PATTERN handling
  nocfbot-0006  Add trailing commas to RPR enum definitions
  nocfbot-0007  Remove optional outer parentheses from ereport() calls in
                RPR files
  nocfbot-0008  Add high-water mark tracking to NFA visited bitmap reset
  nocfbot-0009  Document DEFINE subquery rejection as intentional
                over-rejection
  nocfbot-0010  Remove duplicate #include in nodeWindowAgg.c
  nocfbot-0011  Normalize SQL/RPR standard references
  nocfbot-0012  Add rpr_integration B7 cases for RPR in recursive query
  nocfbot-0013  Reject row pattern recognition in recursive queries
  nocfbot-0014  Enhance README.rpr per Tatsuo Ishii's review
  nocfbot-0015  Round out README.rpr WindowAggState field coverage
  nocfbot-0016  Add raw_expression_tree_walker coverage for RPR raw nodes
  nocfbot-0017  Enhance README.rpr per Jian He's review
  nocfbot-0018  Clarify execRPR.c comments and tighten an Assert
  nocfbot-0019  Change nfa_add_state_unique signature from bool to void
  nocfbot-0020  Add reluctant bounded mid-band test to rpr_nfa
  nocfbot-0021  Define RPR absorption terminology in README.rpr
  nocfbot-0022  Document the get_reduced_frame_status cascade invariant
  nocfbot-0023  Explain the completed-head-context branch in
                update_reduced_frame
  nocfbot-0024  Tighten the RPR frame-boundary check from >= to ==
  nocfbot-0025  Reject single-row window frame in row pattern recognition
  nocfbot-0026  Remove the redundant zero check on the RPR frame ending
                offset

New since the 2026-05-30 post [7] -- nocfbot-0027..0068, in series order.
These apply Jian He's round-3 review [1], an in-house static-analysis
pass, and the v48 cosmetic fold (smallest-first, A..F at the tail).
Patches that change user-visible behavior are tagged [behavior change],
except nocfbot-0042, whose narrow variable-limit change (251 -> 240) is
noted inline.

  nocfbot-0027  Restore findTargetlistEntrySQL99 to static (per Tatsuo)
  nocfbot-0028  Clarify RPR comments
  nocfbot-0029  Rename RPR NFA constructors to make/clone
  nocfbot-0030  Demote RPR nfaVisitedNWords to a local
  nocfbot-0031  Rename the AST-level prefix/suffix rewrite from
                "absorption" to "merging"  (D3)
  nocfbot-0032  Drop non-standard per-group banner labels from RPR
                forward declarations
  nocfbot-0033  Clarify that ExecRPRCleanupDeadContexts always frees the
                failed context
  nocfbot-0034  Correct stale RPR comments and document a defensive
                window check

  nocfbot-0035  Fix unsafe (A{n,})* quantifier flattening
                [behavior change]
      (A{2,})* reaches {0} UNION [2,INF); flattening to A* wrongly admits
      a single A.  Skip multiplication when the outer is skippable and the
      child min >= 2.

  nocfbot-0036  Avoid INF-valued quantifier bound in consecutive-merge
                [behavior change]
      The merge overflow guard used <=, letting a merged min reach the
      RPR_QUANTITY_INF sentinel; tightened to <.

  nocfbot-0037  Fix count slot leak in row pattern recognition absorption
                [behavior change]
      An absorbable leaf VAR exiting inline at max count left its shared
      depth count slot dirty for the sibling group; cleared on exit, with
      the count-clear policy made explicit via Asserts.

  nocfbot-0038  Demote dead runtime checks in the RPR executor to
                assertions

  nocfbot-0039  Fix memory leak in row pattern recognition DEFINE
                evaluation  [behavior change]
      Per-tuple DEFINE evaluation leaked into the wrong context.

  nocfbot-0040  Honor reluctant quantifier for non-leading optional RPR
                variables  [behavior change]
      Reluctant handling existed for leading VARs and groups but not for
      a non-leading optional VAR routed through nfa_route_to_elem; e.g.
      (B A?? C) matched A greedily.  Same skip-first pattern applied.

  nocfbot-0041  Reject column-less compound navigation  [behavior change]
      A compound nav with no column reference is now rejected at parse
      time, instead of being silently accepted with a wrong result.

  nocfbot-0042  Reserve the high varId nibble for RPR control elements
                (the maximum pattern-variable count drops from 251 to 240)

  nocfbot-0043  Generalize quantifier multiplication  [behavior change]
      Replaces the hand-rolled special cases for (child{p,q}){m,n} with a
      general interval-contiguity test, folding cases such as
      (A{2,3}){2,3} -> A{4,9} and (A+){3} -> A{3,} while leaving genuine
      gaps unflattened.

  nocfbot-0044  Tidy up row pattern recognition pattern compilation
  nocfbot-0045  Drop redundant parentheses from row pattern recognition
                ereports
  nocfbot-0046  Tidy up forward declarations and helper placement
  nocfbot-0047  Update the varId documentation
  nocfbot-0048  Reformat the design-decisions chapter in README.rpr
  nocfbot-0049  Point to the absorption-analysis docs from the RPR flag
                definitions

  nocfbot-0050  Fix signed-integer overflow in frame-end clamp
                [behavior change]
      The frame-end clamp computed frameOffset + 1 (int64) before the
      overflow check, so a FOLLOWING offset near PG_INT64_MAX overflowed;
      now added in two checked steps that clamp to PG_INT64_MAX.

  nocfbot-0051  Rework row pattern EXPLAIN deparser to fix grouped
                alternation branches  [EXPLAIN output]
      The bytecode deparser collapsed alternations whose branches were
      quantified groups (e.g. (C | (A B)+ | D)).  Rewritten as recursive
      descent over depth/jump-anchored windows; display-only, execution
      was already correct.

  nocfbot-0052  Handle row pattern navigation nodes in exprTypmod and
                isSimpleNode
  nocfbot-0053  Restore the error cursor for too many row pattern
                variables
  nocfbot-0054  Test deparse of an inline row pattern window
  nocfbot-0055  Fix a mislabeled INITIAL test

  nocfbot-0056  Reject invalid column references in row pattern DEFINE
                clauses  [behavior change]
      Outer-query correlated references and 3-part qualified names in
      DEFINE were reaching an internal "Upper-level Var found where not
      expected" (XX000); both now raise a clean, classified error at
      transformColumnRef.

  nocfbot-0057  Fix shortest match for reluctant nullable quantifiers
                [behavior change]
      An outer reluctant quantifier over a nullable reluctant body, e.g.
      (A??)+?, consumed rows instead of taking the empty match.  The
      count<min branch in nfa_advance_end now mirrors its sibling and
      routes the fast-forward exit first for reluctant elements.

  nocfbot-0058  Compare varno when preserving DEFINE-referenced columns
                [behavior change]
      The allpaths DEFINE-column retention loop compared varattno only;
      it now matches varno/varlevelsup too, dropping over-retention.

  nocfbot-0059  Allow a row pattern quantifier with no space before the
                alternation operator  [behavior change]
      The lexer glues "*|" etc. into one token, so (A*|B) failed.  The
      grammar now recognizes the glued forms via a transient trailing_alt
      flag and splitRPRTrailingAlt() re-splits so "A*|B" == "A* | B".

  nocfbot-0060  Fix outdated function and file references in RPR docs
  nocfbot-0061  Assert that row pattern nesting depth never aliases the
                RPR_DEPTH_NONE sentinel

  nocfbot-0062  Invalidate the row pattern nav slot cache when a window
                partition changes  [behavior change]
      release_partition() now resets nav_slot_pos so a stale cached slot
      position cannot survive a partition boundary.

  nocfbot-0063  Tidy up formatting in row pattern recognition code   (A)
  nocfbot-0064  Modernize idioms in row pattern recognition code     (B)
  nocfbot-0065  Use width-explicit integer limit macros              (C)
                (INT_MAX -> PG_INT32_MAX, the unbounded sentinel)
  nocfbot-0066  Tidy up row pattern recognition regression test comments (D)
  nocfbot-0067  Add row pattern recognition negative and coverage tests (E)
  nocfbot-0068  Use foreach_node and friends in RPR code             (F)
                (foreach + lfirst() -> foreach_node;
                 foreach_current_index, dropping redundant breaks)

Issue-to-patch map (each raised issue -> decision -> landing), by source.
(Earlier incremental cleanup 0001..0016: see [4][5][6].)

Jian He, round 1 (2026-05-26):
  Short-circuit optimization        invited to drive   -> separate series
  Absorption README narrative       accept             -> nocfbot-0017
  DFS expansion                     accept             -> nocfbot-0017
  initialAdvance README mismatch    accept             -> nocfbot-0017
  "absorption" -> "merging" rename  accept (Tatsuo)    -> nocfbot-0031
  Defensive Assert in advance_var   accept             -> nocfbot-0018
  Finalize unnecessary?             keep               -> nocfbot-0018
  Greedy comment label              accept             -> nocfbot-0018
  state->next reset                 decline            -> nocfbot-0018
  visited marking purpose           accept             -> nocfbot-0018
  compareDepth comment              accept             -> nocfbot-0018
  ALT depth invariant Assert        decline            -> nocfbot-0018
  Unused bool return                accept             -> nocfbot-0019
  count >= 3 test coverage          accept             -> nocfbot-0020

Jian He, round 2 (2026-05-28):
  RPRContext consolidation          defer (Tatsuo)     -> D2
  README terminology + typo         accept             -> nocfbot-0021
  get_reduced_frame_status order    keep               -> nocfbot-0022
  states == NULL branch coverage    already reached    -> nocfbot-0023
  currentPos >= -> ==               accept             -> nocfbot-0024
  single-row frame (6.10.2, D1)     reject (Tatsuo)    -> nocfbot-0025
  ExecRPRProcessRow refactor        Datum fix only     -> nocfbot-0026

Jian He, round 3 (2026-05-30) [1] (reply [2]):
  "Tests line N" comments           rephrase           -> nocfbot-0028
  "at the END" wording              accept             -> nocfbot-0028
  uppercase END cleanup             accept             -> nocfbot-0028
  nfa_advance_var count comment     reword             -> nocfbot-0028
  make / clone rename               accept             -> nocfbot-0029
  nfaVisitedNWords field            accept (-> local)  -> nocfbot-0030
  nfaStateSize field                keep               -> (no change)
  drop the elem parameter           keep               -> (no change)

Jian He, off-list (2026-05-15..06-04):
  Per-group banner labels           drop               -> nocfbot-0032
  Dead-context cleanup comment      clarify            -> nocfbot-0033
  rpr.c pg_unreachable refactor     accept             -> nocfbot-0044
  ABSORBABLE doc pointer            accept             -> nocfbot-0049
  has_column_ref necessity          defend             -> (no change)
  CFI dedup                         decline            -> (no change)
  fractional FOLLOWING tests        decline            -> (no change)
  initial/SEEK rename               defer              -> separate series

Tatsuo Ishii + Jian round 4 (2026-05-31..06-02):
  Tatsuo README review              accept             -> nocfbot-0014
  findTargetlistEntrySQL99 static   accept             -> nocfbot-0027
  varId high nibble ("3")           accept             -> nocfbot-0042
  ereport redundant parens          accept             -> nocfbot-0045
  forward-decl / helper placement   accept             -> nocfbot-0046
  varId documentation sync          accept             -> nocfbot-0047
  XII chapter prose reformat        reformat           -> nocfbot-0048
  README 0005 -> 0006 reorder       at submit (Ishii)  -> committer
  double-free question              cannot occur       -> (no change)
  redundant continue                keep               -> (no change)
  non-ASCII section sign            already ASCII      -> (no change)
  empty pattern PATTERN ()          R020 reject right  -> (no change)

in-house static analysis:
  stale comments + window check     fix                -> nocfbot-0034
  unsafe (A{n,})* flattening        fix                -> nocfbot-0035
  INF-valued merge bound            fix                -> nocfbot-0036
  count slot leak                   fix                -> nocfbot-0037
  dead runtime checks -> asserts    fix                -> nocfbot-0038
  DEFINE per-tuple memory leak      fix                -> nocfbot-0039
  reluctant non-leading optional    fix                -> nocfbot-0040
  column-less compound navigation   fix                -> nocfbot-0041
  incomplete quantifier fold (EP-1) generalize         -> nocfbot-0043
  frame-end signed overflow         fix                -> nocfbot-0050
  EXPLAIN deparser grouped-alt      fix                -> nocfbot-0051
  exprTypmod / isSimpleNode nav     fix                -> nocfbot-0052
  errpos for too-many-vars          fix                -> nocfbot-0053
  inline-OVER deparse coverage      fix                -> nocfbot-0054
  INITIAL test mislabel             fix                -> nocfbot-0055
  DEFINE outer / 3-part column ref  fix                -> nocfbot-0056
  reluctant-nullable shortest match fix                -> nocfbot-0057
  allpaths varno comparison         fix                -> nocfbot-0058
  glued quantifier + alternation    fix                -> nocfbot-0059
  nfa_advance / README references   doc                -> nocfbot-0060
  invariant asserts / notes         assert             -> nocfbot-0061
  nav_slot / LAST(x,0) edge cases   defense            -> nocfbot-0062

v48 final fold (off-list sweep + Jian round 5 v47 patch):
  pure formatting                   fold               -> nocfbot-0063
  idioms (palloc_array, Max, ...)   fold               -> nocfbot-0064
  INT_MAX -> PG_INT32_MAX sentinel  fold               -> nocfbot-0065
  regress comment + doc hygiene     fold               -> nocfbot-0066
  new negative / coverage tests     fold               -> nocfbot-0067
  foreach_node + lfirst sweep       fold               -> nocfbot-0068

Please let me know if any of the slicing or the grouping looks off, or
if you would prefer the correctness fixes split out ahead of the
cosmetic fold.

Thanks again to Jian for the careful reading.

Best regards,
Henson

[1] Jian's review, round 3 (2026-05-30):

https://postgr.es/m/CACJufxEsaU8GQ4yeXTWhAO8VjbrZTh5CpvUqz=4a3T0Cwz44pA@mail.gmail.com

[2] My point-by-point reply to the round-3 review (2026-05-30):

https://postgr.es/m/CAAAe_zBi1dOtWb2vnwSvGwuU0-bqAOm_7dOM4u-CmukA8xaV5Q@mail.gmail.com

[3] Tatsuo's note on the 0025 ereport wrapping (2026-06-02):

https://postgr.es/m/20260602.120305.1365406908882385077.ishii@postgresql.org

[4] Earlier incremental post -- nocfbot-0001..0011 (2026-05-09):

https://postgr.es/m/CAAAe_zCL9UtiYthrSaXCmhFMK6Q3YQ6BQGgae7C9en2k=S9doA@mail.gmail.com

[5] Earlier incremental post -- nocfbot-0012..0015 (2026-05-12):

https://postgr.es/m/CAAAe_zBg7y5frYxReua1dczXkMK-7fk6bkVo5ZQXDYxpxe0cwA@mail.gmail.com

[6] nocfbot-0016, sent at the time as 0015 (2026-05-26):

https://postgr.es/m/CAAAe_zD7vCLCb+vxpO3P-NsDUZ=JcN8EGCV0dz0BNgBKsbOGcQ@mail.gmail.com

[7] The 2026-05-30 post -- renumbered 0001..0016 and added 0017..0026:

https://postgr.es/m/CAAAe_zAuHwqUfqJOD4PDUkWsxTfTytNaandq11Kddw2bfCcpvQ@mail.gmail.com

  371. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-09T08:13:07Z

    Hi Henson,

> Note, however, "PATTERN ()" is possible in R010 because it allows to
> omit the DEFINE clause itself.

My mistake. the DEFINE clause is mandatory in R010 too. Sorry for
noise.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  372. Re: Row pattern recognition

    jian he <jian.universality@gmail.com> — 2026-06-09T08:57:20Z

    Hi.

The below in advanced.sgml actually seems better suited for
explain.sgml (which currently has no changes).
At the very least, explain.sgml should mention it.

   <para>
    When examining query plans for Row Pattern Recognition with
    <command>EXPLAIN</command>, the pattern output may include special
    markers that indicate optimization opportunities. A double quote
    <literal>"</literal> marks where pattern absorption can occur,
    and a single quote <literal>'</literal> marks absorbable elements
    within a branch. For example, <literal>a+"</literal> indicates that
    repeated matches of <literal>a</literal> can be absorbed, while
    <literal>(a' b')+"</literal> shows that both <literal>a</literal>
    and <literal>b</literal> within the group are absorbable.
    These markers are primarily useful for understanding internal
    optimization behavior.
   </para>

<programlisting>
......
 PATTERN (LOWPRICE UP+ DOWN+)
 DEFINE
  LOWPRICE AS price &lt;= 100,
  UP AS price &gt; PREV(price),
  DOWN AS price &lt; PREV(price)
);
</programlisting>
LOWPRICE, UP, DOWN should be lower case, since they are not keywords?

scanRPRPatternRecursive
It would be better to replace all  ``non-trivial quantifier`` to
``non-trivial quantifier (not {1,1}).``

I want to rename allocateRPRPattern to makeRPRPattern, what do you think?

RPRPatternNode.reluctant_location is not necessary. we never use it
for error reporting,
and since it's restricted to the end of a pattern, it adds very little value.

Is nav_volatile_func_checker really necessary?
We already have contain_volatile_functions and
contain_volatile_functions_not_nextval.
Also per https://www.postgresql.org/message-id/626986.1776785090@sss.pgh.pa.us
The convention is not to check expression volatility during parse
analysis stage.

+bool        trailing_alt pg_node_attr(query_jumble_ignore);
I may check whether trailing_alt is necessary later,  but
query_jumble_ignore is not needed because it's only manipulated in
gram.y and we do JumbleQuery in parse_analyze_fixedparams, which
obviously is later than all gram.y related work.
----------------------------------------------
v47-0001-v47-rpr-reformat-gram.y-again.nocfbot
The HINT message below is somewhat misleading, since the question mark
(?) is actually after the range quantifier.
Change errmsg to errmsg("invalid token \"%s\" after range quantifier",
$5) will make the error message clearer.

+SELECT FROM rpr_err WINDOW w AS ( ROWS BETWEEN CURRENT ROW AND 1
FOLLOWING PATTERN (A {1,2}??) DEFINE A AS TRUE);
+ERROR:  invalid token "??" after range quantifier
+LINE 1: ...TWEEN CURRENT ROW AND 1 FOLLOWING PATTERN (A {1,2}??) DEFINE...
+                                                             ^
+HINT:  Only "?" is allowed after {n,m} to make it reluctant.
----------------------------------------------
v47-0002-v47-rpr-regress-tests-refactoring.nocfbot

-- Consecutive GROUP merge: (A B){2} (A B)+ -> (a b){3,}
-- Exercises the child->max == RPR_QUANTITY_INF branch in
mergeConsecutiveGroups,
--- where a finite prev ((A B){2,2}) meets an infinite child ((A B)+).

We should just drop these comments from the regress tests.
mergeConsecutiveGroups is a static function that might change, and we don't want
to have to update all the tests every time it does.  Also the comments meaning
is still totally clear without them.
----------------------------------------------
v47-0003-refactor-many-functions-in-rpr.c.nocfbot
I did some major refactoring in rpr.c. Instead of allocating new lists to
remove duplicates, we now just mutate them in place using list_delete_nth_cell
(a common pattern in the codebase).
for overflow handling, it would be better use pg_mul_s32_overflow,
pg_add_s32_overflow.
See the commit message for the details.

v47-0004-another-v47-misc-refactoring.nocfbot
Some misc refactoring, that seems to make sense to me, see the commit
message for rationale.

The above v47-0001, v47-0002, and v47-0003, v47-0004 are based on
https://github.com/assam258-5892/postgres
latest commit.



--
jian
https://www.enterprisedb.com/

  373. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-09T15:15:31Z

    Re: Row pattern recognition
To: jian he, Tatsuo Ishii (+ pgsql-hackers)

Hi jian, Tatsuo,

Thanks for the careful review -- replies inline. One question up front,
for Tatsuo: I'd like your take on the LOWPRICE/UP/DOWN casing below
(details at that point). The rest are replies to jian's points.

> The below in advanced.sgml actually seems better suited for
> explain.sgml (which currently has no changes).
> At the very least, explain.sgml should mention it.

Agreed; I'll move the EXPLAIN marker description over to explain.sgml.

> LOWPRICE, UP, DOWN should be lower case, since they are not keywords?

They're user identifiers, and the standard and Oracle docs show them in
upper case, so I lean toward leaving the example as written -- though
EXPLAIN and deparse do lower-case them today. Tatsuo, I'd defer the
casing to you.

> scanRPRPatternRecursive
> It would be better to replace all ``non-trivial quantifier`` to
> ``non-trivial quantifier (not {1,1}).``

Agreed, that's clearer; will do.

> I want to rename allocateRPRPattern to makeRPRPattern, what do you think?

Good idea -- makeRPRPattern fits the makeNode/makeXxx convention.

> RPRPatternNode.reluctant_location is not necessary.

Agreed. The field is only written and propagated, never read back, so
I'll drop it and keep the reluctance as a plain bool.

> Is nav_volatile_func_checker really necessary? ... The convention is
> not to check expression volatility during parse analysis stage.

Good point -- I'll move the volatility rejection out of parse analysis
and into the planner, in line with that convention.

> query_jumble_ignore is not needed because it's only manipulated in
> gram.y ...

Right; I'll drop the query_jumble_ignore attribute.

> v47-0001 ... Change errmsg to errmsg("invalid token \"%s\" after range
> quantifier", $5) will make the error message clearer.

Agreed, that reads better; I'll adjust the message accordingly.

> v47-0002 ... We should just drop these comments from the regress tests.

Agreed -- the internal function names don't belong in the tests; I'll
remove them.

> v47-0003 ... mutate them in place using list_delete_nth_cell ...
> for overflow handling ... pg_mul_s32_overflow, pg_add_s32_overflow.

The overflow helpers are a clear win. The in-place list mutation is
behavior-equivalent, but I'd rather not reshape that logic right now
without a concrete need, so I'll hold off on it for the moment. The new
tests are good and I'll fold them in.

More broadly, I see the patch as being in its final verification stage
at this point, so I'd prefer to limit logic changes to those backed by a
clear review or test finding.

> v47-0004 ... Some misc refactoring ...

Mostly agree -- the rpSkipTo/initial comments read more accurately as
flag assignments. I'd keep the "PATTERN AST" wording (it's an AST, not a
flag) and the deparse note, but the rest is fine.

Best,
Henson

  374. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-10T06:10:19Z

    Hi Tatsuo, Jian,

While doing a self-review pass over the incremental fixes on top of v47 I
ran into two issues where I'd rather agree on an approach with you before
I pick one.  One of them is a regression I introduced myself in the DEFINE
memory-leak fix; the other is an original design point from v47.  There is
also a third bug which I plan to handle together with the second one, since
it can be affected by that change -- I describe it at the end.

I have verified both of the issues below on an assert-enabled build (and a
non-assert build where relevant).


== 1. DEFINE evaluation reuses the per-output-tuple context
(use-after-free) ==

nocfbot-0039 (the DEFINE memory-leak fix) added a ResetExprContext() in
update_reduced_frame, but it resets the wrong context.

ps_ExprContext is the per-output-tuple context that ExecWindowAgg resets
once per output row.  update_reduced_frame now resets it once per NFA row,
while the output row is still being formed -- so a pass-by-ref window
function result already datum-copied into that per-tuple memory (when
numfuncs > 1) is freed before ExecProject reads it.

Minimal trigger -- a pass-by-ref window function plus a second one over an
RPR window:

  SELECT lag(company) OVER w, count(*) OVER w FROM stock
   WINDOW w AS (PARTITION BY company ORDER BY tdate
                ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
                AFTER MATCH SKIP PAST LAST ROW INITIAL
                PATTERN (START UP+)
                DEFINE START AS TRUE, UP AS price > PREV(price));

On a CLOBBER_FREED_MEMORY build the lag column comes out as 0x7F garbage;
in production it is garbage or a crash.  (An aggregate is not required --
lag + first_value hits the same reset via the frame-access path.)

Neither v47 nor the patch is the answer on its own: v47 had no reset here,
so no use-after-free, but the DEFINE scratch accumulated over the whole
forward scan (the leak nocfbot-0039 fixed); nocfbot-0039 added the per-row
reset but on the shared per-output-tuple context.  We do want a per-row
reset -- just not on that context.

So I think this needs a dedicated ExprContext for DEFINE evaluation, reset
once per NFA row: it keeps the memory bounded without touching the
per-output-tuple results.

Question: does a dedicated DEFINE ExprContext look right to you?


== 2. PREV/NEXT/FIRST/LAST placeholders collide with user functions ==

The nav operations are polymorphic pg_catalog functions (anyelement, OIDs
8126-8133) recognized by funcid in parse_func.c, which collides with
same-name user functions.

Outside DEFINE, a same-name function masks or clashes with the placeholder:
with public.last(anyelement), SELECT last(123) fails "cannot use last
outside a DEFINE clause"; with public.next(numeric), SELECT next(10) fails
"function next(integer) is not unique"; and even with no user function,
last(123) errors instead of "function last(integer) does not exist".

Inside DEFINE, a same-name function with an exact-type match beats the
anyelement placeholder, so PREV(price) silently becomes a plain FuncExpr
instead of an RPRNavExpr -- a wrong match result with no error (reproduced
for numeric, text and int).  And ruleutils deparses a bare PREV(, so
reparsing a view under a search_path with public.prev rebinds it (pg_dump
is safe via search_path = '').

This is original v47 design, not a regression.  Per the standard,
PREV/NEXT/FIRST/LAST are navigation operations with dedicated syntax, not
general-namespace functions -- the collision comes from mapping them onto
catalog functions plus search-path resolution.

I haven't found a clean approach yet.  Inside DEFINE these names have to be
the navigation operation (per the standard), yet outside DEFINE they
shouldn't shadow or break same-name user functions the way the catalog
placeholders do -- and since the deparse output is unqualified (a bare
PREV(...)), whatever we choose also has to round-trip cleanly.  I'm not
sure how best to reconcile those.

My rough leaning is to not add catalog functions for these at all: leave
resolution outside DEFINE exactly as it is today, and only inside DEFINE
adjust the function-resolution path itself to recognize the navigation
operations.  But that is still quite abstract.

Question: how would you approach this?


== Note: a third bug, to be handled together with item 2 ==

A navigation operation nested inside another nav's offset argument -- e.g.
PREV(price, NEXT(2::bigint, 0)) in a DEFINE clause -- slips past the parser
but trips Assert(!IsA(nav->offset_arg, RPRNavExpr)) in the planner.  So it
aborts at plan time on an assert build; without asserts, the backward form
PREV(price, PREV(2::bigint, 5)) reaches a runtime "cannot fetch row ...
before mark position".

The fix is to reject a nav inside an offset argument in the DEFINE walk.
But since item 2 may reshape that walker substantially from how it works
today, I'll do it together with item 2 and add it as a regression test
there.

Thanks,
Henson

  375. Re: Row pattern recognition

    jian he <jian.universality@gmail.com> — 2026-06-10T10:48:01Z

    On Wed, Jun 10, 2026 at 2:10 PM Henson Choi <assam258@gmail.com> wrote:
>
> Hi Tatsuo, Jian,
>
>
> == 2. PREV/NEXT/FIRST/LAST placeholders collide with user functions ==
>
> The nav operations are polymorphic pg_catalog functions (anyelement, OIDs
> 8126-8133) recognized by funcid in parse_func.c, which collides with
> same-name user functions.
>
> Outside DEFINE, a same-name function masks or clashes with the placeholder:
> with public.last(anyelement), SELECT last(123) fails "cannot use last
> outside a DEFINE clause"; with public.next(numeric), SELECT next(10) fails
> "function next(integer) is not unique"; and even with no user function,
> last(123) errors instead of "function last(integer) does not exist".
>
> Inside DEFINE, a same-name function with an exact-type match beats the
> anyelement placeholder, so PREV(price) silently becomes a plain FuncExpr
> instead of an RPRNavExpr -- a wrong match result with no error (reproduced
> for numeric, text and int).  And ruleutils deparses a bare PREV(, so
> reparsing a view under a search_path with public.prev rebinds it (pg_dump
> is safe via search_path = '').
>
> This is original v47 design, not a regression.  Per the standard,
> PREV/NEXT/FIRST/LAST are navigation operations with dedicated syntax, not
> general-namespace functions -- the collision comes from mapping them onto
> catalog functions plus search-path resolution.
>
> I haven't found a clean approach yet.  Inside DEFINE these names have to be
> the navigation operation (per the standard), yet outside DEFINE they
> shouldn't shadow or break same-name user functions the way the catalog
> placeholders do -- and since the deparse output is unqualified (a bare
> PREV(...)), whatever we choose also has to round-trip cleanly.  I'm not
> sure how best to reconcile those.
>
> My rough leaning is to not add catalog functions for these at all: leave
> resolution outside DEFINE exactly as it is today, and only inside DEFINE
> adjust the function-resolution path itself to recognize the navigation
> operations.  But that is still quite abstract.
>
> Question: how would you approach this?
>

SELECT first_value(1);
ERROR:  window function first_value requires an OVER clause
LINE 1: SELECT first_value(1);
               ^

select prosrc, prokind, proname from pg_proc
where proname = 'prev' or proname = 'first' or proname  = 'last' or
proname = 'next';

I am wondering, why the above query result functions not makred as
window function in catalog?

  376. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-10T14:19:02Z

    Hi Henson,

> Hi Tatsuo, Jian,
> 
> While doing a self-review pass over the incremental fixes on top of v47 I
> ran into two issues where I'd rather agree on an approach with you before
> I pick one.  One of them is a regression I introduced myself in the DEFINE
> memory-leak fix; the other is an original design point from v47.  There is
> also a third bug which I plan to handle together with the second one, since
> it can be affected by that change -- I describe it at the end.
> 
> I have verified both of the issues below on an assert-enabled build (and a
> non-assert build where relevant).
> 
> 
> == 1. DEFINE evaluation reuses the per-output-tuple context
> (use-after-free) ==
> 
> nocfbot-0039 (the DEFINE memory-leak fix) added a ResetExprContext() in
> update_reduced_frame, but it resets the wrong context.
> 
> ps_ExprContext is the per-output-tuple context that ExecWindowAgg resets
> once per output row.  update_reduced_frame now resets it once per NFA row,
> while the output row is still being formed -- so a pass-by-ref window
> function result already datum-copied into that per-tuple memory (when
> numfuncs > 1) is freed before ExecProject reads it.
> 
> Minimal trigger -- a pass-by-ref window function plus a second one over an
> RPR window:
> 
>   SELECT lag(company) OVER w, count(*) OVER w FROM stock
>    WINDOW w AS (PARTITION BY company ORDER BY tdate
>                 ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>                 AFTER MATCH SKIP PAST LAST ROW INITIAL
>                 PATTERN (START UP+)
>                 DEFINE START AS TRUE, UP AS price > PREV(price));
> 
> On a CLOBBER_FREED_MEMORY build the lag column comes out as 0x7F garbage;
> in production it is garbage or a crash.  (An aggregate is not required --
> lag + first_value hits the same reset via the frame-access path.)
> 
> Neither v47 nor the patch is the answer on its own: v47 had no reset here,
> so no use-after-free, but the DEFINE scratch accumulated over the whole
> forward scan (the leak nocfbot-0039 fixed); nocfbot-0039 added the per-row
> reset but on the shared per-output-tuple context.  We do want a per-row
> reset -- just not on that context.
> 
> So I think this needs a dedicated ExprContext for DEFINE evaluation, reset
> once per NFA row: it keeps the memory bounded without touching the
> per-output-tuple results.
> 
> Question: does a dedicated DEFINE ExprContext look right to you?

Can we use winstate->tmpcontext instead?
> 
> == 2. PREV/NEXT/FIRST/LAST placeholders collide with user functions ==
> 
> The nav operations are polymorphic pg_catalog functions (anyelement, OIDs
> 8126-8133) recognized by funcid in parse_func.c, which collides with
> same-name user functions.
> 
> Outside DEFINE, a same-name function masks or clashes with the placeholder:
> with public.last(anyelement), SELECT last(123) fails "cannot use last
> outside a DEFINE clause"; with public.next(numeric), SELECT next(10) fails
> "function next(integer) is not unique"; and even with no user function,
> last(123) errors instead of "function last(integer) does not exist".
> 
> Inside DEFINE, a same-name function with an exact-type match beats the
> anyelement placeholder, so PREV(price) silently becomes a plain FuncExpr
> instead of an RPRNavExpr -- a wrong match result with no error (reproduced
> for numeric, text and int).  And ruleutils deparses a bare PREV(, so
> reparsing a view under a search_path with public.prev rebinds it (pg_dump
> is safe via search_path = '').
> 
> This is original v47 design, not a regression.  Per the standard,
> PREV/NEXT/FIRST/LAST are navigation operations with dedicated syntax, not
> general-namespace functions -- the collision comes from mapping them onto
> catalog functions plus search-path resolution.
> 
> I haven't found a clean approach yet.  Inside DEFINE these names have to be
> the navigation operation (per the standard), yet outside DEFINE they
> shouldn't shadow or break same-name user functions the way the catalog
> placeholders do -- and since the deparse output is unqualified (a bare
> PREV(...)), whatever we choose also has to round-trip cleanly.  I'm not
> sure how best to reconcile those.
> 
> My rough leaning is to not add catalog functions for these at all: leave
> resolution outside DEFINE exactly as it is today, and only inside DEFINE
> adjust the function-resolution path itself to recognize the navigation
> operations.  But that is still quite abstract.
> 
> Question: how would you approach this?

I need more time think about this.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  377. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-11T01:20:45Z

    Hi Tatsuo,

== 1. DEFINE evaluation use-after-free (to Tatsuo) ==

>> nocfbot-0039 (the DEFINE memory-leak fix) added a ResetExprContext() in
>> update_reduced_frame, but it resets the wrong context.
>>
>> ps_ExprContext is the per-output-tuple context that ExecWindowAgg resets
>> once per output row.  update_reduced_frame now resets it once per NFA
row,
>> while the output row is still being formed -- so a pass-by-ref window
>> function result already datum-copied into that per-tuple memory (when
>> numfuncs > 1) is freed before ExecProject reads it.
>>
>> Neither v47 nor the patch is the answer on its own: v47 had no reset
here,
>> so no use-after-free, but the DEFINE scratch accumulated over the whole
>> forward scan (the leak nocfbot-0039 fixed); nocfbot-0039 added the
per-row
>> reset but on the shared per-output-tuple context.  We do want a per-row
>> reset -- just not on that context.
>>
>> So I think this needs a dedicated ExprContext for DEFINE evaluation,
reset
>> once per NFA row: it keeps the memory bounded without touching the
>> per-output-tuple results.
>>
>> Question: does a dedicated DEFINE ExprContext look right to you?
>
> Can we use winstate->tmpcontext instead?

I don't think we can.  tmpcontext is idle where update_reduced_frame
would reset it, but not *during* DEFINE evaluation: a DEFINE with NEXT()
re-enters the spooler from inside its ExecEvalExpr (ExecEvalRPRNavSet
-> window_gettupleslot -> spool_tuples), and spool_tuples resets
winstate->tmpcontext (via ExecQualAndReset on partEqfunction) for every
input row it pulls under a PARTITION BY.  ExecEvalRPRNavRestore parks a
pass-by-ref nav result in that per-tuple memory to survive until the
next reset.  To give one example,

  -- the cast forces a pass-by-ref result; a by-value bigint is safe
  DEFINE b AS PREV(price::numeric) < NEXT(price::numeric)

frees PREV's result when NEXT spools the next row, before numeric_lt()
reads it -- the same use-after-free as nocfbot-0039, just on a different
context.  It is the normal forward-scan path (the NFA runs ahead of the
spool), and after a tuplestore spill even NEXT-free DEFINEs hit it.

More generally, tmpcontext fits its existing users because each one sets
its slots, evaluates, and resets within a shallow, straight-line region;
it is best kept to that limited, low-depth usage.  DEFINE evaluation
re-enters the spooler mid-expression, so it cannot honor that contract.

So I think the dedicated DEFINE ExprContext is the right call: a
once-per-NFA-row reset is a third cadence, distinct from tmpcontext
(per-input-tuple) and ps_ExprContext (per-output-tuple) -- the same
one-context-per-cadence pattern nodeWindowAgg and nodeAgg already use.

Thanks,
Henson

  378. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-11T05:05:42Z

    Hi Jian,

On Wed, Jun 10, 2026 jian he <jian.universality@gmail.com> wrote:
> SELECT first_value(1);
> ERROR:  window function first_value requires an OVER clause
> LINE 1: SELECT first_value(1);
>                ^
>
> select prosrc, prokind, proname from pg_proc
> where proname = 'prev' or proname = 'first' or proname  = 'last' or
> proname = 'next';
>
> I am wondering, why the above query result functions not marked as
> window function in catalog?

The short answer is that no catalog marking can fix this, because prokind
does not take part in name resolution.  func_get_detail() picks the
candidate by name/arguments/search_path first, and prokind (window or not)
is only inspected afterwards.  So marking the placeholders as window
functions would leave all three collisions in place: next(10) still
ambiguous against public.next(numeric), an exact-signature
public.prev(integer) still silently winning inside DEFINE, and the
deparsed bare PREV( still rebinding under an ordinary search_path.

The deeper reason is that whether these names are navigation operations
depends on a context the catalog cannot see -- are we inside a DEFINE
clause or not?  Any approach that goes through the catalog, under any
prokind, resolves the name before it can know whether it is even a
navigation operation.

So the only thing that actually works is to not rely on the catalog at all:
inside a DEFINE clause, recognize these names before any catalog lookup and
substitute the navigation operation directly.  Concretely I plan to drop
the eight catalog entries (OIDs 8126-8133) and, inside DEFINE, intercept an
unqualified prev/next/first/last by name in ParseFuncOrColumn -- ahead of
func_get_detail() -- binding it to the navigation operation with no
fallback to function resolution.  Outside DEFINE the names then resolve as
ordinary identifiers (a same-name user function just works), and a
schema-qualified call inside DEFINE (public.prev(...)) still reaches the
user function as the explicit escape hatch.  This is also what Oracle does
-- there is no catalog object for the navigation operations there at all.

Thanks for the pointer -- the first_value comparison is what made it clear
that the robustness comes from context, not from prokind.

Henson

  379. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-11T10:29:48Z

    Hi hackers,

This refreshes the v48 series [1]: Jian He's review of the v47-000x
cleanup patches [2] (my point-by-point reply [3]) is now applied as
nocfbot-0069..0077.  nocfbot-0001..0068 are unchanged from the last post
[1] (rebase only), except nocfbot-0039, which is voided here (see below).

Resolved since the last post:

  Jian He, round 5 -- the v47-0001..0004 patches plus the inline comments
  [2].  Applied across nocfbot-0069..0077 (issue map below).

For Tatsuo -- two calls I'd like from you before this settles:

  1. nocfbot-0073 moves the DEFINE volatility rejection out of parse
     analysis into the planner, per the convention Jian and Tom noted.  Two
     things to weigh: it is a small behavior change -- a volatile DEFINE
     hidden in a view is now rejected when the view is read, not at CREATE
     VIEW -- and the planner has no ParseState, so the error cursor is
     reconstructed from debug_query_string, a first at the optimizer stage.
     If either gives you pause, I'll rework it or split nocfbot-0073 out for
     separate review ahead of the cosmetic patches.

  2. LOWPRICE/UP/DOWN casing.  I've left the variables upper-case (the
     standard and Oracle show them so), but EXPLAIN and deparse lower-case
     them today, so the examples and the actual output disagree.  If you'd
     prefer the section lower-cased, I'll do it as a doc patch.

Deferred:

  v47-0003 -- in its final verification stage I'm keeping logic changes to
      those backed by a clear review or test finding.  The two new
      group-merge tests are folded in (nocfbot-0076); the in-place
      list_delete_nth_cell reshaping and the pg_mul_s32_overflow guard are
      held out and, absent a concrete finding, will go in as a separate
      patch rather than this fold.

On cfbot:

  The SIGILL cfbot reports against this entry is not from RPR.  After Andres
  Freund noted cfbot failing inside the JIT on this patch [4], I found that
  any JIT-compiled query crashes the same way on the CI's libLLVM 19.1 +
ASAN
  build; it reproduces on plain master with a trivial aggregate, landing in
  libLLVM's decodeDiscriminator.  I reported it as its own issue with a
  standalone reproducer [5]; its CF entry is only there to surface the
  problem on cfbot, and I'll withdraw it once the community has taken note.
  Not a blocker for this series.

Attached: the v47 feature series (v47-0001..0009) rebased onto current
master, plus the full incremental series (nocfbot-0001..0077) on top.

Base:

  9d141466ff0  2026-06-10  Undo thinko in commit e78d1d6d4.

Already posted (nocfbot-0001..0068): unchanged since v48 [1], rebase only,
except nocfbot-0039 (voided; no patch).  Titles for reference:

  nocfbot-0001  Add DEFINE non-volatile baseline to rpr_integration B9
  nocfbot-0002  Unify RPR DEFINE walkers and reject volatile callees
  nocfbot-0003  Cover RPR empty-match path with EXPLAIN tests; fix stale
                XXX comments
  nocfbot-0004  Reclassify DEFINE qualifier check and reword diagnostic to
                "expression"
  nocfbot-0005  Sync stale comments on DEFINE/PATTERN handling
  nocfbot-0006  Add trailing commas to RPR enum definitions
  nocfbot-0007  Remove optional outer parentheses from ereport() calls in
                RPR files
  nocfbot-0008  Add high-water mark tracking to NFA visited bitmap reset
  nocfbot-0009  Document DEFINE subquery rejection as intentional
                over-rejection
  nocfbot-0010  Remove duplicate #include in nodeWindowAgg.c
  nocfbot-0011  Normalize SQL/RPR standard references
  nocfbot-0012  Add rpr_integration B7 cases for RPR in recursive query
  nocfbot-0013  Reject row pattern recognition in recursive queries
  nocfbot-0014  Enhance README.rpr per Tatsuo Ishii's review
  nocfbot-0015  Round out README.rpr WindowAggState field coverage
  nocfbot-0016  Add raw_expression_tree_walker coverage for RPR raw nodes
  nocfbot-0017  Enhance README.rpr per Jian He's review
  nocfbot-0018  Clarify execRPR.c comments and tighten an Assert
  nocfbot-0019  Change nfa_add_state_unique signature from bool to void
  nocfbot-0020  Add reluctant bounded mid-band test to rpr_nfa
  nocfbot-0021  Define RPR absorption terminology in README.rpr
  nocfbot-0022  Document the get_reduced_frame_status cascade invariant
  nocfbot-0023  Explain the completed-head-context branch in
                update_reduced_frame
  nocfbot-0024  Tighten the RPR frame-boundary check from >= to ==
  nocfbot-0025  Reject single-row window frame in row pattern recognition
  nocfbot-0026  Remove the redundant zero check on the RPR frame ending
                offset
  nocfbot-0027  Restore findTargetlistEntrySQL99 to static (per Tatsuo)
  nocfbot-0028  Clarify RPR comments
  nocfbot-0029  Rename RPR NFA constructors to make/clone
  nocfbot-0030  Demote RPR nfaVisitedNWords to a local
  nocfbot-0031  Rename the AST-level prefix/suffix rewrite from
                "absorption" to "merging"
  nocfbot-0032  Drop non-standard per-group banner labels from RPR
                forward declarations
  nocfbot-0033  Clarify that ExecRPRCleanupDeadContexts always frees the
                failed context
  nocfbot-0034  Correct stale RPR comments and document a defensive
                window check
  nocfbot-0035  Fix unsafe (A{n,})* quantifier flattening
  nocfbot-0036  Avoid INF-valued quantifier bound in consecutive-merge
  nocfbot-0037  Fix count slot leak in row pattern recognition absorption
  nocfbot-0038  Demote dead runtime checks in the RPR executor to
                assertions
  nocfbot-0039  (voided -- no patch; the v48 DEFINE memory-leak fix reset
                the wrong context and is backed out here, being re-fixed on
                a separate thread)
  nocfbot-0040  Honor reluctant quantifier for non-leading optional RPR
                variables
  nocfbot-0041  Reject column-less compound navigation
  nocfbot-0042  Reserve the high varId nibble for RPR control elements
  nocfbot-0043  Generalize quantifier multiplication
  nocfbot-0044  Tidy up row pattern recognition pattern compilation
  nocfbot-0045  Drop redundant parentheses from row pattern recognition
                ereports
  nocfbot-0046  Tidy up forward declarations and helper placement
  nocfbot-0047  Update the varId documentation
  nocfbot-0048  Reformat the design-decisions chapter in README.rpr
  nocfbot-0049  Point to the absorption-analysis docs from the RPR flag
                definitions
  nocfbot-0050  Fix signed-integer overflow in frame-end clamp
  nocfbot-0051  Rework row pattern EXPLAIN deparser to fix grouped
                alternation branches
  nocfbot-0052  Handle row pattern navigation nodes in exprTypmod and
                isSimpleNode
  nocfbot-0053  Restore the error cursor for too many row pattern variables
  nocfbot-0054  Test deparse of an inline row pattern window
  nocfbot-0055  Fix a mislabeled INITIAL test
  nocfbot-0056  Reject invalid column references in row pattern DEFINE
                clauses
  nocfbot-0057  Fix shortest match for reluctant nullable quantifiers
  nocfbot-0058  Compare varno when preserving DEFINE-referenced columns
  nocfbot-0059  Allow a row pattern quantifier with no space before the
                alternation operator
  nocfbot-0060  Fix outdated function and file references in RPR docs
  nocfbot-0061  Assert that row pattern nesting depth never aliases the
                RPR_DEPTH_NONE sentinel
  nocfbot-0062  Invalidate the row pattern nav slot cache when a window
                partition changes
  nocfbot-0063  Tidy up formatting in row pattern recognition code
  nocfbot-0064  Modernize idioms in row pattern recognition code
  nocfbot-0065  Use width-explicit integer limit macros
  nocfbot-0066  Tidy up row pattern recognition regression test comments
  nocfbot-0067  Add row pattern recognition negative and coverage tests
  nocfbot-0068  Use foreach_node and friends in RPR code

New since v48 [1] -- nocfbot-0069..0077, in series order.  These apply
Jian He's round-5 review [2] (issue map below).  Behavior-changing patches
are tagged [behavior change].

  nocfbot-0069  Move RPR EXPLAIN marker description and add a worked
                example
      The double-quote (") and single-quote (') absorption-marker
      description moves out of the advanced.sgml tutorial into perform.sgml
      "Using EXPLAIN" -- where PG documents EXPLAIN output markers -- with a
      worked example showing Pattern: (up' down')+".

  nocfbot-0070  Clarify the non-trivial quantifier comments in row pattern
                recognition
      All inline "non-trivial quantifier" comments now read "non-trivial
      quantifier (not {1,1}).", matching the form already used in the
      function header.

  nocfbot-0071  Rename allocateRPRPattern to makeRPRPattern in row pattern
                recognition
      Matches the makeNode/makeRPR* convention; buildRPRPattern keeps its
      name (make = low-level allocation, build = higher-level compilation).

  nocfbot-0072  Remove the unused reluctant_location field from
                RPRPatternNode
      The field was only written and propagated, never read back; the
      reluctance stays as a plain bool.

  nocfbot-0073  Reject volatile DEFINE expressions in the planner, not at
                parse time  [behavior change]
      Moves the DEFINE volatility rejection out of parse analysis into the
      planner.  The DEFINE walker keeps only the RPR-specific nav checks.
      See the note to Tatsuo above (view-read timing; debug_query_string
      error cursor).

  nocfbot-0074  Drop query_jumble_ignore from the RPR trailing_alt field
      splitRPRTrailingAlt finalizes trailing_alt to false before
      JumbleQuery runs, so the attribute was a no-op.

  nocfbot-0075  Quote the offending token in RPR quantifier syntax errors
      v47-0001: the offending token is now quoted -- invalid token "%s"
      after range quantifier, and likewise after "*"/"+" quantifier.  (Known
      gap: the "invalid quantifier combination" else-branch, e.g. A? ??,
      still doesn't quote its token -- follow-up.)

  nocfbot-0076  Clean up RPR regression test comments and add group-merge
                tests
      v47-0002: internal static function names dropped from the regress
      comments.  Plus the two new group-merge tests from v47-0003 (cannot-
      merge prefix; 5-way prefix+suffix), rebuilt against the current code
      so the expected output matches.

  nocfbot-0077  Improve comment accuracy in RPR parse analysis and tests
      v47-0004: rpSkipTo/initial comments reworded as flag assignments,
      keeping the "PATTERN AST" wording (it is an AST, not a flag) and the
      deparse note.

Issue-to-patch map (Jian He, round 5 [2] -> decision -> landing):

  EXPLAIN marker -> explain.sgml     accept (perform.sgml)  -> nocfbot-0069
  LOWPRICE/UP/DOWN casing            defer (Tatsuo)         -> (no change)
  non-trivial quantifier (not {1,1}) accept                 -> nocfbot-0070
  allocateRPRPattern -> make rename  accept                 -> nocfbot-0071
  reluctant_location field           accept (remove)        -> nocfbot-0072
  nav_volatile_func_checker          move to planner        -> nocfbot-0073
  query_jumble_ignore on trailing_alt accept (remove)       -> nocfbot-0074
  v47-0001 quote offending token     accept                 -> nocfbot-0075
  v47-0002 drop fn names in comments accept                 -> nocfbot-0076
  v47-0003 new merge tests           accept                 -> nocfbot-0076
  v47-0003 in-place + overflow guard hold (verification)    -> (no change)
  v47-0004 misc comment accuracy     accept (keep AST)      -> nocfbot-0077

Still to come, each tracked on the thread where I raised it and to land in
a later fold once its approach is agreed:

  - The DEFINE-evaluation memory leak (nocfbot-0039's interim fix turned it
    into a use-after-free, so it is voided here); the proper fix is a
    dedicated DEFINE ExprContext that resolves both.
  - The PREV/NEXT/FIRST/LAST namespace collision (the same change also
    rejects the nested-nav-in-offset case).

Quality work to run alongside, once those land: Valgrind (leak and
use-after-free) and standing gcov coverage to catch untested planner paths,
on Linux.  Independently of the cfbot JIT issue, I also intend to review the
RPR spool's tuplestore handling -- mark position and trimming in particular.

Longer term, and out of scope for this CF entry:

  - Smaller documentation and error-message follow-ups (glossary, the
    bounded-quantifier message, README wording).
  - Short-circuit / tri-state DEFINE evaluation, as a separate series.
  - SEEK clause support (SQL:2016).
  - Empty pattern PATTERN () -- correctly rejected under R020 today, and
    meaningful only once R010 lands.
  - Relaxing the DEFINE-subquery over-rejection where the standard permits
    it (the subquery does no RPR of its own and makes no outer reference).
  - Prefix-pattern absorption (an optimization; designed and intentionally
    split out of v47 as its own series).
  - R010 (MATCH_RECOGNIZE in the FROM clause) and the shared RPRContext it
    would back.

Please let me know if any of the slicing or grouping looks off.

Thanks again to Jian for the careful reading.

Best regards,
Henson

[1] The v48 fold (2026-06-09):

https://postgr.es/m/CAAAe_zCd1pA6vCaMD7e3hB3Ou+=mZziB1=CO_tBybuAiE5K=vQ@mail.gmail.com

[2] Jian's review of the v47-000x patches (2026-06-09):

https://postgr.es/m/CACJufxFnwdQSApt2vWwYCd0gtf+JjFDxT2hbxHi=+dhFJc+-1g@mail.gmail.com

[3] My point-by-point reply (2026-06-09):

https://postgr.es/m/CAAAe_zATnkqsbLYDj8MJV1TriX9Wi0wShDg3nK3qYpiupKwhFA@mail.gmail.com

[4] Andres Freund's note that cfbot was failing inside the JIT on this
    patch:

https://postgr.es/m/p7r5bekdbl2zcazid7agvfo2nfnq5bim2a5jkckqygld32n325@fctfp6ou6qnb

[5] LLVM JIT: any JIT-compiled query crashes (SIGILL) on a libLLVM 19 + ASAN
    build (2026-06-10; CF entry 6870):

https://postgr.es/m/CAAAe_zD6jGANGZFKnHLKHF8izqmqqJbVe=NOuERFwN_Spj5VOA@mail.gmail.com

  380. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-12T03:40:50Z

    Hi Henson,

> Hi Tatsuo,
> 
> == 1. DEFINE evaluation use-after-free (to Tatsuo) ==
> 
>>> nocfbot-0039 (the DEFINE memory-leak fix) added a ResetExprContext() in
>>> update_reduced_frame, but it resets the wrong context.
>>>
>>> ps_ExprContext is the per-output-tuple context that ExecWindowAgg resets
>>> once per output row.  update_reduced_frame now resets it once per NFA
> row,
>>> while the output row is still being formed -- so a pass-by-ref window
>>> function result already datum-copied into that per-tuple memory (when
>>> numfuncs > 1) is freed before ExecProject reads it.
>>>
>>> Neither v47 nor the patch is the answer on its own: v47 had no reset
> here,
>>> so no use-after-free, but the DEFINE scratch accumulated over the whole
>>> forward scan (the leak nocfbot-0039 fixed); nocfbot-0039 added the
> per-row
>>> reset but on the shared per-output-tuple context.  We do want a per-row
>>> reset -- just not on that context.
>>>
>>> So I think this needs a dedicated ExprContext for DEFINE evaluation,
> reset
>>> once per NFA row: it keeps the memory bounded without touching the
>>> per-output-tuple results.
>>>
>>> Question: does a dedicated DEFINE ExprContext look right to you?
>>
>> Can we use winstate->tmpcontext instead?
> 
> I don't think we can.  tmpcontext is idle where update_reduced_frame
> would reset it, but not *during* DEFINE evaluation: a DEFINE with NEXT()
> re-enters the spooler from inside its ExecEvalExpr (ExecEvalRPRNavSet
> -> window_gettupleslot -> spool_tuples), and spool_tuples resets
> winstate->tmpcontext (via ExecQualAndReset on partEqfunction) for every
> input row it pulls under a PARTITION BY.  ExecEvalRPRNavRestore parks a
> pass-by-ref nav result in that per-tuple memory to survive until the
> next reset.  To give one example,
> 
>   -- the cast forces a pass-by-ref result; a by-value bigint is safe
>   DEFINE b AS PREV(price::numeric) < NEXT(price::numeric)
> 
> frees PREV's result when NEXT spools the next row, before numeric_lt()
> reads it -- the same use-after-free as nocfbot-0039, just on a different
> context.  It is the normal forward-scan path (the NFA runs ahead of the
> spool), and after a tuplestore spill even NEXT-free DEFINEs hit it.
> 
> More generally, tmpcontext fits its existing users because each one sets
> its slots, evaluates, and resets within a shallow, straight-line region;
> it is best kept to that limited, low-depth usage.  DEFINE evaluation
> re-enters the spooler mid-expression, so it cannot honor that contract.
> 
> So I think the dedicated DEFINE ExprContext is the right call: a
> once-per-NFA-row reset is a third cadence, distinct from tmpcontext
> (per-input-tuple) and ps_ExprContext (per-output-tuple) -- the same
> one-context-per-cadence pattern nodeWindowAgg and nodeAgg already use.

Thanks for the explanation. Got it. We need the third ExprContext.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  381. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-12T04:03:25Z

    Hi Henson,

> Hi hackers,
> 
> This refreshes the v48 series [1]: Jian He's review of the v47-000x
> cleanup patches [2] (my point-by-point reply [3]) is now applied as
> nocfbot-0069..0077.  nocfbot-0001..0068 are unchanged from the last post
> [1] (rebase only), except nocfbot-0039, which is voided here (see below).
> 
> Resolved since the last post:
> 
>   Jian He, round 5 -- the v47-0001..0004 patches plus the inline comments
>   [2].  Applied across nocfbot-0069..0077 (issue map below).
> 
> For Tatsuo -- two calls I'd like from you before this settles:

Ok, let me check...

>   1. nocfbot-0073 moves the DEFINE volatility rejection out of parse
>      analysis into the planner, per the convention Jian and Tom noted.  Two
>      things to weigh: it is a small behavior change -- a volatile DEFINE
>      hidden in a view is now rejected when the view is read, not at CREATE
>      VIEW -- and the planner has no ParseState, so the error cursor is
>      reconstructed from debug_query_string, a first at the optimizer stage.
>      If either gives you pause, I'll rework it or split nocfbot-0073 out for
>      separate review ahead of the cosmetic patches.

No pause from me.

>   2. LOWPRICE/UP/DOWN casing.  I've left the variables upper-case (the
>      standard and Oracle show them so), but EXPLAIN and deparse lower-case
>      them today, so the examples and the actual output disagree.  If you'd
>      prefer the section lower-cased, I'll do it as a doc patch.

I see no problem here. PostgreSQL always convert identifiers into
lower case. Evenrybody knows that.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  382. Re: Row pattern recognition

    jian he <jian.universality@gmail.com> — 2026-06-12T10:52:52Z

    Hi.
Based on https://github.com/assam258-5892/postgres/commits/RPR
Here are more minor review comments:

v47-0001-replace-some-AST-to-clause.nocfbot
Replace some "AST" to "clause".

v47-0002-refactor-gram.y.nocfbot
group Op and trailing ALT into one IF condition, this might improve readability.

v47-0003-refactor-volatile-expression-check.nocfbot

validate_rpr_define_volatility function, along with its helper
functions is removed.
contain_volatile_functions() already covers both volatile FuncExpr callees and
NextValueExpr, so a separate walker is not needed. The trade-off is that we lose
the error cursor position, but that seems better than to maintaining extra code.

The check is also moved to after preprocess_expression(), following the same
pattern as contain_volatile_functions_after_planning.

v47-0004-refactor-validateRPRPatternVarCount.nocfbot

validateRPRPatternVarCount() is recursive. Using the `rpDefs != NULL` sentinel
to gate the "DEFINE variable not used in PATTERN" check at the outermost call
level is awkward. That cross-check only needs to run once; it is better
expressed in the caller, transformDefineClause().

v47-0005-refactor-transformDefineClause.nocfbot

coerce_to_boolean() was deferred to a second pass over the completed
defineClause list, meaning pull_var_clause() ran on an expression that had not
yet been fully coerced/transformed. The more intuitively order should be:
transformExpr → coerce_to_boolean → pull_var_clause, so pull_var_clause always
sees the final expression form.

v47-0006-refactor-transformDefineClause.nocfbot

The allowed nav-expression combination is limited, therefore,
has_column_ref is not necessary.
Column reference checks can use contain_var_clause, and it's cheap.

Also, I made the following error message change:

-ERROR:  row pattern navigation offset must be a run-time constant
+ERROR:  row pattern navigation offset expression must not contain
column references



--
jian
https://www.enterprisedb.com/

  383. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-13T02:48:31Z

    Hi Henson,

I looked into 0051 and have some suggestions.

If you like, can you please included the changes in v49?
I am going to release v48 soon.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

> index 1a754bcdac5..7ba0b6df849 100644
> --- a/src/backend/commands/explain.c
> +++ b/src/backend/commands/explain.c

> +static int
> +deparse_rpr_node(RPRPattern *pattern, int i, int limit, StringInfo buf)

One letter function argument name "i" looks unusual and better to be
named more meaningful one, like idx?

>  /*
> - * Process an ALT element: adjust depth parens and register separator positions.
> + * Find the END that closes the group opened by the BEGIN at beginIdx: the
> + * first END at the same depth scanning forward.
>   */
> -static void
> -deparse_rpr_alt(RPRPattern *pattern, int *idx, StringInfoData *buf,
> -				RPRDepth *prevDepth, bool *needSpace, List **altSeps)
> +static int
> +rpr_match_end(RPRPattern *pattern, int beginIdx)
>  {
> -	RPRPatternElement *elem = &pattern->elements[*idx];
> -
> -	/* Close parens for depth decrease */
> -	while (*prevDepth > elem->depth)
> -	{
> -		appendStringInfoChar(buf, ')');
> -		(*prevDepth)--;
> -		*needSpace = true;
> -	}
> -
> -	/* Open parens up to ALT's depth */
> -	while (*prevDepth < elem->depth)
> -	{
> -		if (*needSpace)
> -			appendStringInfoChar(buf, ' ');
> -		appendStringInfoChar(buf, '(');
> -		(*prevDepth)++;
> -		*needSpace = false;
> -	}
> +	RPRDepth	d = pattern->elements[beginIdx].depth;
> +	int			j;

If we change function arugument to other than "i", "j" here better to
be named "i".

>  /*
> - * Process a VAR element: adjust depth parens and output variable name.
> + * Scope end of the construct at index i: the first following element whose
> + * depth is no greater than i's own.  For an ALT marker this is the index just
> + * past its last branch, since depth stays constant across branch boundaries.
> + * FIN sits at depth 0, so a top-level ALT stops there.
>   */
> -static void
> -deparse_rpr_var(RPRPattern *pattern, int *idx, StringInfoData *buf,
> -				RPRDepth *prevDepth, bool *needSpace, List **altSeps)
> +static int
> +rpr_alt_scope_end(RPRPattern *pattern, int i)

One letter argument name "i" looks unusual and better to be named more
meaningful one, like idx?

>  {
> -	RPRPatternElement *elem = &pattern->elements[*idx];
> -
> -	/* Open parens for depth increase */
> -	while (*prevDepth < elem->depth)
> -	{
> -		if (*needSpace)
> -			appendStringInfoChar(buf, ' ');
> -		appendStringInfoChar(buf, '(');
> -		(*prevDepth)++;
> -		*needSpace = false;
> -	}
> +	RPRDepth	d = pattern->elements[i].depth;
> +	int			k;

If we change function arugument to other than "i", "j" here better to
be named "i".

> +/*
> + * Boundary of the alternation branch starting at b (i.e. the start of the next
> + * branch, or altEnd if b is the last branch).
> + *
> + * The branch-start element's jump points at the next branch when this is not
> + * the last branch.  jump is overloaded (a group BEGIN also uses it for its
> + * skip path), so confirm a real branch boundary with the relative test
> + * elem[j-1].next != j: at a true boundary the preceding branch's tail has its
> + * next redirected past the alternation, so it does not point at j.
> + */
> +static int
> +rpr_next_branch(RPRPattern *pattern, int b, int altEnd)

One letter function argument name "b" looks unusual and better to be
named more meaningful one, like bno or bid?

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  384. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-13T12:25:30Z

    Hi,

> Hi hackers,
> 
> This refreshes the v48 series [1]: Jian He's review of the v47-000x
> cleanup patches [2] (my point-by-point reply [3]) is now applied as
> nocfbot-0069..0077.  nocfbot-0001..0068 are unchanged from the last post
> [1] (rebase only), except nocfbot-0039, which is voided here (see below).

I have created v48 RPR (Row pattern recognition) patches based on the
attached incremental patches against v47 patches. I have briefly
looked through all the incremental patches. If I have missed
something, I will follow up later.  The patches are based on the
commit 3e3d7875e95 on the master branch.

The series of patches are to implement the row pattern recognition
(SQL/RPR) feature. Currently the implementation is a subset of SQL/RPR
(ISO/IEC 19075-2:2016). Namely, implementation of some features of
R020 (WINDOW clause). R010 (MATCH_RECOGNIZE) is out of the scope of
the patches.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  385. Re: Row pattern recognition

    jian he <jian.universality@gmail.com> — 2026-06-14T02:39:42Z

    On Sat, Jun 13, 2026 at 8:26 PM Tatsuo Ishii <ishii@postgresql.org> wrote:
>
> I have created v48 RPR (Row pattern recognition) patches based on the
> attached incremental patches against v47 patches. I have briefly
> looked through all the incremental patches. If I have missed
> something, I will follow up later.  The patches are based on the
> commit 3e3d7875e95 on the master branch.
>

get_windowclause_startup_tuples
do we need some comments for WindowClause->defineClause

src/backend/executor/README.rpr
  (2) Transcription to WindowClause
      - Copies rpPattern, rpSkipTo, initial fields

In transformRPR, we did this:
``wc->rpPattern = windef->rpCommonSyntax->rpPattern;``
Since this is an assignment, not a copy, is "Copies rpPattern" incorrect?

ISO reference typo: most occurrences use {ISO/IEC 19075},
but rpr_integration.sql and some places reference 9075-2:2016 instead of 19075.
We may need check all occurrences to ensure consistency.

In struct ResultRelInfo, we have fields like {ri_CheckConstraintExprs,
ri_WithCheckOptionExprs}.
Similarly, I want to rename WindowAggState->defineClauseList to
defineClauseExprs

I think the comment on function contain_rpr_walker can be removed,
since transformWithClause already explains it.

Is "judgment" really the right word?

I don't fully understand the purpose of eval_nav_offset_helper now.
Also, the branch where (isnull == true) and (offset < 0) appears
unreachable; it's never hit in the regression tests.
We should add some regression tests for these scenarios.



--
jian
https://www.enterprisedb.com/

  386. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-14T07:19:24Z

    > get_windowclause_startup_tuples
> do we need some comments for WindowClause->defineClause

We already have comments for DEFINE clause. Do you think it's not enough?

	 * Moreover, if row pattern recognition is used, we charge the DEFINE
	 * expressions once per tuple for each variable that appears in PATTERN.

> src/backend/executor/README.rpr
>   (2) Transcription to WindowClause
>       - Copies rpPattern, rpSkipTo, initial fields
> 
> In transformRPR, we did this:
> ``wc->rpPattern = windef->rpCommonSyntax->rpPattern;``
> Since this is an assignment, not a copy, is "Copies rpPattern" incorrect?

I think "Copies" is Ok here. Yes, we do not use copyObject to copy the
rpPattern object but copying just the pointer can be regarded as kind
of "copy".

> ISO reference typo: most occurrences use {ISO/IEC 19075},
> but rpr_integration.sql and some places reference 9075-2:2016 instead of 19075.
> We may need check all occurrences to ensure consistency.

19075 and 9075-2 are the different standard documents.

19075-5: Information technology ― Guidance for the use of database language SQL ― Part 5: Row pattern recognition
9075-2: Information technology ― Database languages ― SQL ― Part 2: Foundation (SQL/Foundation)

IMO they are used differently depending on the context.

> In struct ResultRelInfo, we have fields like {ri_CheckConstraintExprs,
> ri_WithCheckOptionExprs}.
> Similarly, I want to rename WindowAggState->defineClauseList to
> defineClauseExprs

Looks reasonable proposal to me.

> I think the comment on function contain_rpr_walker can be removed,
> since transformWithClause already explains it.

I don't think we want to remove all function comments of
contain_rpr_walker. However I am not against removing "Used by
transformWithClause() to enforce ISO/IEC 9075-2:2016 7.17 SR 3)f) on
WITH RECURSIVE elements." because it's somewhat redundant.

> Is "judgment" really the right word?
> 
> I don't fully understand the purpose of eval_nav_offset_helper now.
> Also, the branch where (isnull == true) and (offset < 0) appears
> unreachable; it's never hit in the regression tests.
> We should add some regression tests for these scenarios.

Yeah. Maybe we can turn them into Assertions?

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  387. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-15T01:56:12Z

    Hi jian,

Thanks for the follow-up review patches. Once v48 is posted I'll base the
work on it and submit the accepted ones in v49. Reasoning per patch below,
with the disposition summarized at the end.

> v47-0001 Replace some "AST" to "clause".

Agreed on dropping "AST" -- it isn't our house term; "parse tree" is the
idiom across the tree, and "AST" barely shows up outside these RPR files
(just a jsonpath comment). One tweak: where the text names the data
structure (the README diagram and III-2 header, the parsenodes.h comments
on rpPattern), I'd use "PATTERN parse tree" rather than "clause", since
"clause" loses the tree-vs-flat-NFA contrast. Plain prose that just means
the SQL clause -> "clause", as you have it.

> v47-0002 group Op and trailing ALT into one IF condition, this might
> improve readability.

Here I'd lean toward keeping the current form. The complexity here comes
from the lexer (regex operators tokenized by the SQL operator lexer), and
grouping the branches doesn't remove it -- it just layers more cleverness
on top. I'd rather not add code complexity onto the lexer complexity; the
flat one-token-one-branch form stays easy to scan. A real simplification
would be lexer-side, out of scope here, so I'd leave 0002 as-is unless you
feel strongly.

> v47-0003 validate_rpr_define_volatility ... is removed.
> contain_volatile_functions() already covers both volatile FuncExpr
> callees and NextValueExpr ... The trade-off is that we lose the error
> cursor position, but that seems better than maintaining extra code.

You're right the detection is equivalent. But this restructures the
parse-side define_walker, which the upcoming PREV/NEXT name-binding fix
also reworks -- it even removes the spot where this check incidentally
backstops a nav mis-binding. I'd rather make the simplify-vs-keep call
against that final walker shape than churn it twice, so I'll keep the
current planner-side check for now and revisit once that fix lands.

> v47-0004 validateRPRPatternVarCount() ... the `rpDefs != NULL`
> sentinel ... is awkward. That cross-check only needs to run once;
> better expressed in the caller, transformDefineClause().

Agreed. The cross-check is a one-shot, list-level test -- a different kind
of thing from the recursive per-node count, and not really what
validateRPRPatternVarCount is for (validating the pattern var count).
Moving it to the caller runs it once by construction and leaves the
function true to its name, a pure count. Rejection behavior is unchanged.

> v47-0005 ... The more intuitive order should be: transformExpr ->
> coerce_to_boolean -> pull_var_clause, so pull_var_clause always sees
> the final expression form.

Agreed, that order is more intuitive. I'll make the change and watch for
any change in error output or behavior -- coerce_to_boolean now runs
per-define before pull_var_clause rather than in a second pass -- with
guard tests for domain-over-bool, nav Var preserved, and non-coercible
error, and submit it in v49 if those and the regression pass clean.

> v47-0006 ... has_column_ref is not necessary. Column reference checks
> can use contain_var_clause, and it's cheap. Also the message change:
> -row pattern navigation offset must be a run-time constant
> +row pattern navigation offset expression must not contain column
>  references

Agreed contain_var_clause is simpler and the new message reads better.
But this restructures define_walker too, which the nav name-binding fix
also reworks -- so as with 0003 I'd rather settle it against the post-fix
walker shape than churn it twice, and revisit once that fix lands.

Summary:

  0001  -> v49   (use "parse tree" where it names the structure, not
"clause")
  0002  -> keep the current flat form
  0003  -> revisit after the nav name-binding fix
  0004  -> v49
  0005  -> v49   (after guard tests + regression pass)
  0006  -> revisit after the nav name-binding fix

For 0003 and 0006 -- both swap a hand-rolled walker for a standard helper
(contain_volatile_functions, contain_var_clause) -- I'll think this
through properly when I rework the walker: the convention of reaching for
those helpers against what the current checks buy us (the error cursor,
the specific messages), rather than deciding it by reflex.

More broadly, the substantive correctness defects take priority: the nav
name-binding fix and the rest of that series come ahead of these cleanup
refactors. Once v48 is posted I'll write up the current list of known
issues and send it to the thread.

Thanks again for the careful pass.

Best,
Henson

  388. Re: Row pattern recognition

    jian he <jian.universality@gmail.com> — 2026-06-15T06:32:01Z

    Hi.

More review based on https://github.com/assam258-5892/postgres/commits/RPR

remove_unused_subquery_outputs
``````
                        if (dvar->varno == var->varno &&
                            dvar->varattno == var->varattno &&
                            dvar->varlevelsup == var->varlevelsup)
                        {
                            needed_by_define = true;
                            break;
                        }
``````
dvar, var both can be whole-row Vars, but this seems to work for whole-row vars.
We need some simple regress tests for cases where both are whole-row
vars or one of them is a whole-row var.

I've attached a patch with some refactoring. The below is detailed
commit message.
--------------------------------------------------------------------
RPRNavExpr->resulttype should also marked as pg_node_attr(query_jumble_ignore)

collectPatternVariables is not needed.
The parser already ensures every DEFINE variable appears in PATTERN,
so there is nothing to filter.
Also, we don't really do anything special (like make a dummy Const) regarding
PATTERN variables that not appearing in the DEFINE clause.
See nfa_evaluate_row the for loop break.

buildDefineVariableList is trivial. No need to export it as an
external function.

Rename WindowAggState.defineClauseList to defineClauseExprs
Minor refactoring of regress test comments.
Flatten a needlessly nested block in show_window_def().
Replace a post-loop ListCell NULL check in
remove_unused_subquery_outputs() with a boolean flag.
Reduce the number of arguments in make_windowagg.
--------------------------------------------------------------------



--
jian
https://www.enterprisedb.com/

  389. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-16T05:32:58Z

    Hi Tatsuo, Jian,

I think there's a correctness problem in the RPR patch: a window function's
result can change depending on which other, unrelated window functions are
in the same query.

Pattern matching only advances when a window function reads the frame.
nth_value(x, n) returns NULL without reading the frame when n is NULL
(correct per the standard), so if it is the only window function in an RPR
window, the match never advances over those rows and the reduced frame no
longer matches a full scan.

Example -- one partition, 60 rows, price = id * 10:

  CREATE TABLE rpr_dormant (id int, price int);
  INSERT INTO rpr_dormant SELECT g, g*10 FROM generate_series(1,60) g;

  SELECT id, nth_value(price, CASE WHEN id < 50 THEN NULL ELSE 1 END) OVER w
  FROM rpr_dormant
  WINDOW w AS (
    ORDER BY id
    ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
    AFTER MATCH SKIP PAST LAST ROW
    PATTERN (A+)
    DEFINE A AS price > PREV(FIRST(price), 50)
  );

Run alone, the nth_value column does not follow the actual match structure;
adding an unrelated first_value(id) OVER w, which reads the frame every row,
changes it.  And while the match is dormant the mark position keeps
advancing, running ahead and trimming rows that the backward navigation
later needs -- so the same query can instead fail with "cannot fetch row N
before WindowObject's mark position".

I think an RPR window should perform the match for each row up front,
building its reduced frame during the row scan before the window functions
are evaluated, regardless of whether any function reads the frame.  The fix
belongs in the executor, not in nth_value -- the early return is standard,
and the same gap is reachable from any user-defined function that skips the
frame.

Does this direction seem right, or is the lazy, frame-driven matching
intentional in a way I'm missing?  Happy to prepare a patch.

Best regards,
Henson

  390. Re: Row pattern recognition

    jian he <jian.universality@gmail.com> — 2026-06-17T05:14:09Z

    On Mon, Jun 15, 2026 at 2:32 PM jian he <jian.universality@gmail.com> wrote:
>
> Hi.
>
> More review based on https://github.com/assam258-5892/postgres/commits/RPR
>

/* arity: a value expression and an optional offset */
Typo: arity

I've attached a patch with some refactoring.
Please note that the inline comments have not yet been updated to
match this refactoring.
The below is detailed commit message.
------------------------------------------------------------------------------------------------
Simplify ParseFuncOrColumn:
It now routes to ParseRPRNavCall exclusively when ParseExprKind is
EXPR_KIND_RPR_DEFINE *and* not column projection *and*
list_length(funcname) == 1.
Original behavior is preserved otherwise.

Centralize error handling:
Treat RPR navigation as FUNCDETAIL_NORMAL to reuse the common error handling in
ParseFuncOrColumn, effectively stripping redundant error checks from
ParseRPRNavCall.

Other miscellaneous code cleanups and minor refactoring.
------------------------------------------------------------------------------------------------



--
jian
https://www.enterprisedb.com/

  391. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-17T07:33:26Z

    Hi Jian and Tatsuo,

Thanks for the patch and the careful review.

Tatsuo, item 1 below (attribute notation inside a DEFINE clause) is a
question for you; the rest is feedback on Jian's patch.

> /* arity: a value expression and an optional offset */
> Typo: arity

"arity" may be an unfamiliar term, so I'll reword the comment in plainer
language ("takes a value expression and an optional offset").

> Simplify ParseFuncOrColumn:
> It now routes to ParseRPRNavCall exclusively when ParseExprKind is
> EXPR_KIND_RPR_DEFINE and not column projection and list_length(funcname)
> == 1. Original behavior is preserved otherwise.
> Centralize error handling:
> Treat RPR navigation as FUNCDETAIL_NORMAL to reuse the common error
> handling in ParseFuncOrColumn, effectively stripping redundant error
> checks from ParseRPRNavCall.

I'd take this structural part -- it's a clear cleanup: ParseRPRNavCall
drops the duplicated decoration checks while the common path gives the
identical messages, with no change in behavior or output.

Two user-visible changes in the patch I'd rather settle on their own
before taking them:

1. Attribute notation inside a DEFINE clause, e.g. (f).prev.

   The guard this change removes is one I deliberately left undecided
   during development (hence the XXX comment), so I'd keep it for now and
   ask here.  Without it, (f).prev with no such field gives a generic
   "column \"prev\" not found ..." instead of the dedicated "cannot use
   row pattern navigation function PREV in attribute notation".  Three
   options:

   (a) Treat (f).prev as an ordinary function (prev(f)), the same as
       outside a DEFINE clause -- which is what the patch does.

   (b) Treat (f).prev as the navigation function -- read the attribute
       notation as navigation.  An ordinary function of that name is still
       reachable as public.prev(...).

   (c) Reject the ambiguous (f).prev with a dedicated error (what is
       currently committed), rather than resolving it one way or the
       other.

   My own leaning is actually (a) -- it keeps attribute notation behaving
   the same inside and outside a DEFINE clause.  (c) is what's in the tree
   now, and either way it changes the user-visible error and SQLSTATE, so
   I'd rather settle this explicitly than let the refactor decide it
   silently.  Tatsuo, what do you think?

2. The offset type-mismatch message.

   The patch rephrases

     offset argument of %s must be type bigint, not type %s
   to
     %s offset argument of type %s cannot be coerced to the expected
     bigint   (+ a hint)

   The behavior is identical, so I'd keep the original wording.  "argument
   ... must be type X, not type Y" is the established phrasing -- it's what
   coerce_to_boolean() produces, and the non-boolean DEFINE error in this
   same feature already uses it.  Rewording only the offset message would
   be inconsistent with that, for no behavioral gain.

One small process note: for patches that aren't really being proposed to
the list yet -- ones still under review, or throwaway implementations
written just to analyze a problem -- could you send them off-list or as a
pull request instead?  That keeps the thread focused on what's actually
being proposed for the patch.

Best regards,
Henson

  392. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-17T13:13:27Z

    Hi Henson,

> Hi Tatsuo, Jian,
> 
> I think there's a correctness problem in the RPR patch: a window function's
> result can change depending on which other, unrelated window functions are
> in the same query.
> 
> Pattern matching only advances when a window function reads the frame.
> nth_value(x, n) returns NULL without reading the frame when n is NULL
> (correct per the standard), so if it is the only window function in an RPR
> window, the match never advances over those rows and the reduced frame no
> longer matches a full scan.

Ouch.

> Example -- one partition, 60 rows, price = id * 10:
> 
>   CREATE TABLE rpr_dormant (id int, price int);
>   INSERT INTO rpr_dormant SELECT g, g*10 FROM generate_series(1,60) g;
> 
>   SELECT id, nth_value(price, CASE WHEN id < 50 THEN NULL ELSE 1 END) OVER w
>   FROM rpr_dormant
>   WINDOW w AS (
>     ORDER BY id
>     ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>     AFTER MATCH SKIP PAST LAST ROW
>     PATTERN (A+)
>     DEFINE A AS price > PREV(FIRST(price), 50)
>   );
> 
> Run alone, the nth_value column does not follow the actual match structure;
> adding an unrelated first_value(id) OVER w, which reads the frame every row,
> changes it.  And while the match is dormant the mark position keeps
> advancing, running ahead and trimming rows that the backward navigation
> later needs -- so the same query can instead fail with "cannot fetch row N
> before WindowObject's mark position".
> 
> I think an RPR window should perform the match for each row up front,
> building its reduced frame during the row scan before the window functions
> are evaluated, regardless of whether any function reads the frame.  The fix
> belongs in the executor, not in nth_value -- the early return is standard,
> and the same gap is reachable from any user-defined function that skips the
> frame.
> 
> Does this direction seem right, or is the lazy, frame-driven matching
> intentional in a way I'm missing?  Happy to prepare a patch.

Yes, I think the direction is correct. Probably the patch would someting like this?

diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index cb6a484b7de..b6c12096c85 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -2523,6 +2523,9 @@ ExecWindowAgg(PlanState *pstate)
 			{
 				if (winstate->rpSkipTo == ST_NEXT_ROW)
 					clear_reduced_frame(winstate);
+
+				update_reduced_frame(winstate->nav_winobj,
+									 winstate->frameheadpos);
 			}
 
 			/*


> is the lazy, frame-driven matching
> intentional in a way I'm missing?

Not intentional.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  393. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-18T07:48:54Z

    Hi Jian,

Thanks for the misc-refactoring patch (v48-0001-v48-misc-refactoring) and
the
whole-row Var note.  As before, now that v48 is posted I'll rebase on it
and send
these as incremental patches on top (v49); per-item disposition below, with
the
per-commit breakdown to come with the patch.


> RPRNavExpr->resulttype should also marked as
pg_node_attr(query_jumble_ignore)

Agreed -- every other derived result-type field in primnodes.h already
carries it,
and resulttype is derived from the (jumbled) arg type, so ignoring it loses
nothing.
I'll add it.


> collectPatternVariables is not needed.
> The parser already ensures every DEFINE variable appears in PATTERN ...
> See nfa_evaluate_row the for loop break.

Confirmed -- the parser already rejects a DEFINE variable not used in
PATTERN, so
every DEFINE var is in PATTERN; filtering is redundant and cost_windowagg
can
iterate defineClause directly, so I'll remove it.


> buildDefineVariableList is trivial. No need to export it as an external
function.

Agreed -- I'll drop it and inline the list-building into
create_windowagg_plan().


> Rename WindowAggState.defineClauseList to defineClauseExprs

Agreed (the elements are ExprState, so it matches the ...Exprs convention)
-- I'll
do it, including the additional site the dormant-match fix touches.


> Flatten a needlessly nested block in show_window_def().
> Replace a post-loop ListCell NULL check in
remove_unused_subquery_outputs()
> with a boolean flag.
> Reduce the number of arguments in make_windowagg.
> Minor refactoring of regress test comments.

I'll do all of these.

One exception: the parsenodes.h RPRPatternNode comment names the data
structure, so
rather than "clause" I'll change it to "parse tree node".


> dvar, var both can be whole-row Vars, but this seems to work for
whole-row vars.
> We need some simple regress tests for cases where both are whole-row vars
or one
> of them is a whole-row var.

Good catch -- I'll add regress coverage for the whole-row Var cases (both
sides
whole-row, and one side whole-row) in the needed_by_define check, in v49.

I'll send the items above as a first incremental patch on top of v48, and
take up
the later reviews after that.


Thanks,
Henson

  394. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-18T09:50:07Z

    Hi Henson,

> Hi Jian and Tatsuo,
> 
> Thanks for the patch and the careful review.
> 
> Tatsuo, item 1 below (attribute notation inside a DEFINE clause) is a
> question for you; the rest is feedback on Jian's patch.

> 1. Attribute notation inside a DEFINE clause, e.g. (f).prev.
> 
>    The guard this change removes is one I deliberately left undecided
>    during development (hence the XXX comment), so I'd keep it for now and
>    ask here.  Without it, (f).prev with no such field gives a generic
>    "column \"prev\" not found ..." instead of the dedicated "cannot use
>    row pattern navigation function PREV in attribute notation".  Three
>    options:
> 
>    (a) Treat (f).prev as an ordinary function (prev(f)), the same as
>        outside a DEFINE clause -- which is what the patch does.
> 
>    (b) Treat (f).prev as the navigation function -- read the attribute
>        notation as navigation.  An ordinary function of that name is still
>        reachable as public.prev(...).
> 
>    (c) Reject the ambiguous (f).prev with a dedicated error (what is
>        currently committed), rather than resolving it one way or the
>        other.
> 
>    My own leaning is actually (a) -- it keeps attribute notation behaving
>    the same inside and outside a DEFINE clause.  (c) is what's in the tree
>    now, and either way it changes the user-visible error and SQLSTATE, so
>    I'd rather settle this explicitly than let the refactor decide it
>    silently.  Tatsuo, what do you think?

I think either (a) or (c) is fine. (b) gives no clear benefit to
users. Who want to write (f).prev? Also (f, 10).prev is a syntax
error, which confuses users.  If choosing (a) makes our code cleaner
and more logical, I have no reason to against it.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  395. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-18T10:43:51Z

    Hi Tatsuo,

Thanks -- I'll go in this direction; your sketch has the shape of it.

> Yes, I think the direction is correct. Probably the patch would someting
> like this?
>
> + update_reduced_frame(winstate->nav_winobj,
> + winstate->frameheadpos);

Right: drive the match once per row, up front, instead of letting frame
access drive it.  That's the plan, with two refinements on top of the
sketch.

First, a small refactor: I'll pull the reduced-frame loop and the mark
advance out into their own functions, called once per row before the window
functions run -- so the match tracks the row scan rather than frame reads.

Second, the mark: I'll advance it per NFA row from the match frontier rather
than the output row, so the tuplestore is trimmed as soon as the match is
done with each row.

Thanks,
Henson

  396. Re: Row pattern recognition

    jian he <jian.universality@gmail.com> — 2026-06-19T00:07:53Z

    On Thu, Jun 18, 2026 at 5:50 PM Tatsuo Ishii <ishii@postgresql.org> wrote:
>
> Hi Henson,
>
> > Hi Jian and Tatsuo,
> >
> > Thanks for the patch and the careful review.
> >
> > Tatsuo, item 1 below (attribute notation inside a DEFINE clause) is a
> > question for you; the rest is feedback on Jian's patch.
>
> > 1. Attribute notation inside a DEFINE clause, e.g. (f).prev.
> >
> >    The guard this change removes is one I deliberately left undecided
> >    during development (hence the XXX comment), so I'd keep it for now and
> >    ask here.  Without it, (f).prev with no such field gives a generic
> >    "column \"prev\" not found ..." instead of the dedicated "cannot use
> >    row pattern navigation function PREV in attribute notation".  Three
> >    options:
> >
> >    (a) Treat (f).prev as an ordinary function (prev(f)), the same as
> >        outside a DEFINE clause -- which is what the patch does.
> >
> >    (b) Treat (f).prev as the navigation function -- read the attribute
> >        notation as navigation.  An ordinary function of that name is still
> >        reachable as public.prev(...).
> >
> >    (c) Reject the ambiguous (f).prev with a dedicated error (what is
> >        currently committed), rather than resolving it one way or the
> >        other.
> >
> >    My own leaning is actually (a) -- it keeps attribute notation behaving
> >    the same inside and outside a DEFINE clause.  (c) is what's in the tree
> >    now, and either way it changes the user-visible error and SQLSTATE, so
> >    I'd rather settle this explicitly than let the refactor decide it
> >    silently.  Tatsuo, what do you think?
>

ParseFuncOrColumn cleanly handles (f).prev by translating it to prev(f) as a
regular function call. However, if a dedicated window navigation function
exists, this translation creates ambiguity — it becomes unclear whether prev(f)
is window navigation or a normal function call.

Cases with additional dots (e.g., public.prev(arg)) should also be
treated as normal
function calls, IMHO.

As a result, only prev(arg) and prev(arg, offset) are recognized as special
window navigation syntax, despite being visually identical to a function call.

Summary:
Dedicated window navigation functions should be removed entirely.  Window
navigation should be limited to a single syntactic form (no dots) — one that
*looks* like a function call but is parsed as syntax.

This is not unprecedented; there are many existing cases where something appears
to be a function call but is actually a syntax form, for example:

SELECT json_object('{}');
 json_object
-------------
 {}
(1 row)

SELECT public.json_object('{}');
ERROR:  function public.json_object(unknown) does not exist
LINE 1: SELECT public.json_object('{}');
               ^

So I think in the DEFINE context, it makes sense for some form that
looks like a function call to actually be syntax.

  397. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-19T00:43:10Z

    Hi Jian,

> ParseFuncOrColumn cleanly handles (f).prev by translating it to prev(f)
as a
> regular function call. However, if a dedicated window navigation function
> exists, this translation creates ambiguity -- it becomes unclear whether
> prev(f) is window navigation or a normal function call.

Agreed, and that is the reason I would rather not have a dedicated
navigation function at all.  With navigation handled purely as syntax,
(f).prev never has to compete with it: attribute notation just falls
through to an ordinary function (prev(f)), the same inside and outside a
DEFINE clause.  That is Tatsuo's and my preferred option (a), so I will
settle on it.

> Cases with additional dots (e.g., public.prev(arg)) should also be treated
> as normal function calls, IMHO.

Yes.  A schema-qualified name is the explicit escape hatch to an ordinary
function; navigation is recognized only for an unqualified, single-element
name.

> As a result, only prev(arg) and prev(arg, offset) are recognized as
special
> window navigation syntax, despite being visually identical to a function
> call.

Right -- that is exactly the surface I want to land on: the dotless
prev(...)/next(...)/first(...)/last(...) forms are navigation, everything
else is an ordinary function.

> Summary:
> Dedicated window navigation functions should be removed entirely.  Window
> navigation should be limited to a single syntactic form (no dots) -- one
> that *looks* like a function call but is parsed as syntax.

That is the direction I will take the tree.  So the three of us have
converged.

> This is not unprecedented; there are many existing cases where something
> appears to be a function call but is actually a syntax form, for example:
>
> SELECT json_object('{}');
>  json_object
> -------------
>  {}
> (1 row)
>
> SELECT public.json_object('{}');
> ERROR:  function public.json_object(unknown) does not exist
> LINE 1: SELECT public.json_object('{}');
>                ^
>
> So I think in the DEFINE context, it makes sense for some form that looks
> like a function call to actually be syntax.

The json_object parallel is exact at the user-visible level, but I want to
flag that the implementation has to differ underneath -- and that is
actually why navigation is not done the json_object way.

json_object can live in the grammar because two constraints do not apply to
it:

  - No context restriction.  JSON_OBJECT means the same thing wherever a
    value expression is allowed, so a single keyword production in the
    shared a_expr grammar is enough.

  - The name is safe to reserve.  Making json_object a keyword costs almost
    nobody an identifier.

Row pattern navigation has both constraints:

  - It must mean navigation *only* inside DEFINE; everywhere else
    prev/next/first/last are ordinary names.  bison is LALR(1) and
    context-free, so a production cannot be conditioned on "are we inside
    DEFINE"; a_expr is shared by SELECT lists, WHERE, etc.  And a row
    pattern definition is "ColId AS a_expr", so navigation can appear
    anywhere a value expression can.  Special-casing it in the grammar
    would mean duplicating the whole a_expr tree into a second DEFINE-only
    expression grammar (or resorting to lexer feedback), which is not worth
    it.

  - The names are common.  prev/next/first/last (especially next, first,
    last) are everyday column and function names; reserving them globally
    would break existing queries.

So the plan is the opposite of a grammar keyword: parse navigation as an
ordinary FuncCall and reinterpret it in parse analysis, gated on
p_expr_kind == EXPR_KIND_RPR_DEFINE and an unqualified, single-element
name.  That keeps one expression grammar, leaves the names free everywhere
else, and confines the special meaning to exactly the DEFINE context -- the
same "looks like a function call, parsed as syntax" behavior you described,
reached by a lighter path that does not grow the keyword list.

Thanks,
Henson

  398. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-19T06:10:54Z

    Hi hackers,

When a WINDOW clause is not referenced by any window function, the pattern
is never matched and the DEFINE expressions are never evaluated.

One consequence is that a DEFINE which would raise a run-time error raises
nothing when the window is unused.  The same window definition behaves
differently depending only on whether a window function consumes it:

  CREATE TABLE t (id int, v int);
  INSERT INTO t VALUES (1, 10), (2, 20), (3, 15);

-- (1) the window IS used (count(*) OVER w): the DEFINE is evaluated

  SELECT count(*) OVER w AS cnt
  FROM t
  WINDOW w AS (
      ORDER BY id
      ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
      INITIAL
      PATTERN (A+)
      DEFINE A AS (1 / (v - v)) > 0
  );
  -- ERROR:  division by zero

-- (2) the window is NOT used (no window function): same definition

  SELECT v
  FROM t
  WINDOW w AS (
      ORDER BY id
      ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
      INITIAL
      PATTERN (A+)
      DEFINE A AS (1 / (v - v)) > 0
  );
  --  v
  -- ----
  --  10
  --  20
  --  15
  -- (no error)

EXPLAIN (VERBOSE, COSTS OFF) of (2) confirms the window, and with it the
DEFINE expression, are gone entirely:

      Seq Scan on public.t
        Output: v, id

(A column-dependent division by zero is used so that the expression does not
constant-fold and can only error at run time.)

This follows from existing, general planner behavior.  A window definition
that no window function references is removed from the plan by
select_active_windows(); the planner does this for every window, because a
window with no consumer produces no output, so sorting or partitioning it
would be wasted work.  Before RPR that reasoning was complete: an ordinary
window has nothing but its output, so dropping an unused one changes nothing
observable.

RPR is what turns this into a question.  A DEFINE clause is a per-row
predicate that can have a run-time effect of its own -- here, raising an
error -- independent of any output the window produces.  So the assumption
behind the optimization, that an unused window has no observable behavior,
no
longer holds automatically once a window carries a DEFINE.  Whether the
existing "drop it" optimization should still extend to an RPR window is a
decision RPR's addition forces, not something the prior behavior settles.

The question is whether the current behavior is what we want:

  (a) Keep it.  Skipping the pattern matching for a window that produces
      nothing is the natural optimization, and an expression that is never
      evaluated raising no error is normal behavior.

  (b) Run the matching anyway, even though there is no output to produce, so
      that DEFINE errors are raised regardless of whether a window function
      consumes the window.

Option (a) is cheaper and consistent with how unevaluated expressions
behave, but I lean towards (b): a faulty DEFINE should fail consistently
rather than pass silently just because no window function happens to consume
the window.  I would like to hear what you think.

Regards,
Henson

  399. Re: Row pattern recognition

    jian he <jian.universality@gmail.com> — 2026-06-19T06:21:24Z

    Hi.

CREATE TABLE stock (company TEXT, tdate DATE, price INTEGER);
CREATE TEMP TABLE stock (company TEXT, tdate DATE, price INTEGER);
SELECT count(*) over w FROM stock WINDOW w AS ( ROWS BETWEEN CURRENT
ROW AND UNBOUNDED FOLLOWING PATTERN (A) DEFINE A AS
pg_temp.stock.price > 0 );
SELECT count(*) over w FROM stock WINDOW w AS ( ROWS BETWEEN CURRENT
ROW AND UNBOUNDED FOLLOWING PATTERN (A) DEFINE A AS public.stock.price
> 0 );
SELECT count(*) over w FROM stock WINDOW w AS ( ROWS BETWEEN CURRENT
ROW AND UNBOUNDED FOLLOWING PATTERN (A) DEFINE A AS stock.price > 0 );

The error messages for the above 3 SELECT queries are different.
(pg_temp.stock.price, public.stock.price, stock.price) mean the same
thing: column reference,
Should we try to make the error messages consistent?

ERROR:  range variable qualified expression "rpr_composite.items" is
not allowed in DEFINE clause

"Range variable qualified expression" is non-standard that may confuse users.
To improve clarity and consistency, let's align this with the
established error pattern:

ERROR: invalid reference to FROM-clause entry for table "the_table"



What do you all think about renaming validateRPRPatternVarCount to
preprocessRPRPattern?
--------------------------------------------------------------------------------------------------
v48 [1] has a conflict with master, so the attached patches are based
on https://github.com/assam258-5892/postgres/commits/RPR

The attached v48-0001 mainly replaces PG_INT32_MAX with
RPR_QUANTITY_INF, but it also includes other changes.
See the commit message for detail.

The attached v48-0002 patch is more about miscellaneous refactoring.
The commit message is included below, enclosed by "-----":
--------------------------------------------------------------------------------------------------
Refactor buildRPRPattern to accept a WindowClause pointer directly. This
eliminates the need to pass internal fields like rpPattern, rpSkipTo, and
frameOptions as separate arguments.

collectDefineVariables is not needed, it's simple, and can integrated its logic
directly into buildRPRPattern easily.  Also removed tryUnwrapSingleChild since
its was simple enough to inline inside optimizeSeqPattern and
optimizeAltPattern.

Slightly adjust variable limits error message.

Restructure multiple INSERT INTO ... VALUES statements into multi-value insert
blocks (e.g., INSERT INTO ... VALUES (), ()). This is for brevity and is a good
practice even though performance gains is minor.

Long PATTERN strings (more than 2000 width!) in the regression tests were broken
into formatted multiline queries for readability.
--------------------------------------------------------------------------------------------------
[1] https://postgr.es/m/20260613.212530.63949290085162247.ishii%40postgresql.org



--
jian
https://www.enterprisedb.com/

  400. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-19T08:57:28Z

    Hi Henson,

> Hi hackers,
> 
> When a WINDOW clause is not referenced by any window function, the pattern
> is never matched and the DEFINE expressions are never evaluated.
> 
> One consequence is that a DEFINE which would raise a run-time error raises
> nothing when the window is unused.  The same window definition behaves
> differently depending only on whether a window function consumes it:
> 
>   CREATE TABLE t (id int, v int);
>   INSERT INTO t VALUES (1, 10), (2, 20), (3, 15);
> 
> -- (1) the window IS used (count(*) OVER w): the DEFINE is evaluated
> 
>   SELECT count(*) OVER w AS cnt
>   FROM t
>   WINDOW w AS (
>       ORDER BY id
>       ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>       INITIAL
>       PATTERN (A+)
>       DEFINE A AS (1 / (v - v)) > 0
>   );
>   -- ERROR:  division by zero
> 
> -- (2) the window is NOT used (no window function): same definition
> 
>   SELECT v
>   FROM t
>   WINDOW w AS (
>       ORDER BY id
>       ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
>       INITIAL
>       PATTERN (A+)
>       DEFINE A AS (1 / (v - v)) > 0
>   );
>   --  v
>   -- ----
>   --  10
>   --  20
>   --  15
>   -- (no error)
> 
> EXPLAIN (VERBOSE, COSTS OFF) of (2) confirms the window, and with it the
> DEFINE expression, are gone entirely:
> 
>       Seq Scan on public.t
>         Output: v, id
> 
> (A column-dependent division by zero is used so that the expression does not
> constant-fold and can only error at run time.)
> 
> This follows from existing, general planner behavior.  A window definition
> that no window function references is removed from the plan by
> select_active_windows(); the planner does this for every window, because a
> window with no consumer produces no output, so sorting or partitioning it
> would be wasted work.  Before RPR that reasoning was complete: an ordinary
> window has nothing but its output, so dropping an unused one changes nothing
> observable.
> 
> RPR is what turns this into a question.  A DEFINE clause is a per-row
> predicate that can have a run-time effect of its own -- here, raising an
> error -- independent of any output the window produces.  So the assumption
> behind the optimization, that an unused window has no observable behavior,
> no
> longer holds automatically once a window carries a DEFINE.  Whether the
> existing "drop it" optimization should still extend to an RPR window is a
> decision RPR's addition forces, not something the prior behavior settles.
> 
> The question is whether the current behavior is what we want:
> 
>   (a) Keep it.  Skipping the pattern matching for a window that produces
>       nothing is the natural optimization, and an expression that is never
>       evaluated raising no error is normal behavior.
> 
>   (b) Run the matching anyway, even though there is no output to produce, so
>       that DEFINE errors are raised regardless of whether a window function
>       consumes the window.
> 
> Option (a) is cheaper and consistent with how unevaluated expressions
> behave, but I lean towards (b): a faulty DEFINE should fail consistently
> rather than pass silently just because no window function happens to consume
> the window.  I would like to hear what you think.

I think we should follow (a).

From ISO/IEC 9075-2:2016 7.15 <window clause> General Rules:
------------------------------------
1) Let SL be the <select list> of the <query specification> or <select
statement: single row> that immediately contains TE [1].

Case:

a) If SL does not simply contain a <window function>, then the <window
clause> is disregarded, and the result of TE is the result of the last
<from clause>, <where clause>, <group by clause> or <having clause> of
TE.
------------------------------------
[1] TE: Table expression

So I think the standard requires a window clause to be disregarded if
window function is not included in the select list. As DEFINE is a
part of a window clause, it should be disregarded if there's no window
function in the window clause too.

> behave, but I lean towards (b): a faulty DEFINE should fail consistently
> rather than pass silently just because no window function happens to consume
> the window.  I would like to hear what you think.

But we already pass faulty window clauses. Example:

-- If window function exists, faulty window clause (invalid frame
-- ending offset) is detected.
PREPARE prep AS
SELECT count(*) OVER w
FROM generate_series(1,5) g(i)
WINDOW w AS (
       ROWS BETWEEN CURRENT ROW AND $1 FOLLOWING
);
PREPARE
EXECUTE prep(-1);
psql:prepare.sql:7: ERROR:  frame ending offset must not be negative

DEALLOCATE prep;
DEALLOCATE

-- But if window function does not exist, the faulty window clause
-- (invalid frame ending offset) is not detected.
PREPARE prep AS
SELECT i
FROM generate_series(1,5) g(i)
WINDOW w AS (
       ROWS BETWEEN CURRENT ROW AND $1 FOLLOWING
);
PREPARE
EXECUTE prep(-1);
 i 
---
 1
 2
 3
 4
 5
(5 rows)

I think if we detect faulty DEFINE in the last case , it's not only
against the standard but against our existing behavior.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  401. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-19T11:39:01Z

    Hi,

Attached is the v49 RPR patches. Just rebased. Except that, nothing
has been changed since v48.

> I have created v48 RPR (Row pattern recognition) patches based on the
> attached incremental patches against v47 patches. I have briefly
> looked through all the incremental patches. If I have missed
> something, I will follow up later.  The patches are based on the
> commit 3e3d7875e95 on the master branch.
> 
> The series of patches are to implement the row pattern recognition
> (SQL/RPR) feature. Currently the implementation is a subset of SQL/RPR
> (ISO/IEC 19075-2:2016). Namely, implementation of some features of
> R020 (WINDOW clause). R010 (MATCH_RECOGNIZE) is out of the scope of
> the patches.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  402. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-19T12:08:19Z

    Hi Tatsuo,

> From ISO/IEC 9075-2:2016 7.15 <window clause> General Rules:
> ...
> a) If SL does not simply contain a <window function>, then the <window
> clause> is disregarded, and the result of TE is the result of the last
> <from clause>, <where clause>, <group by clause> or <having clause> of TE.

Thank you -- this is exactly the text I was missing.  I only have ISO/IEC
19075-5 (the guidance / technical report), which describes what a window
definition does but says nothing about a window clause being disregarded
when no window function references it.  The General Rule you quote settles
it: disregarding an unused window clause is required by the standard, and
since DEFINE is part of the window clause, it is disregarded along with the
rest.  So evaluating its DEFINE -- option (b) -- would mean running a clause
the standard tells us to ignore.

> But we already pass faulty window clauses. Example:
> ... ERROR:  frame ending offset must not be negative ...
> I think if we detect faulty DEFINE in the last case, it's not only
> against the standard but against our existing behavior.

Agreed, and the parallel is convincing.  An unused window already escapes
the negative-frame-offset check, so singling out DEFINE as the one clause
that must fire regardless would be inconsistent with how the rest of an
unused window already behaves.

This also matches the planner: select_active_windows() drops any window with
no referencing WindowFunc, and its comment already cites the same <window
clause> General Rules (General Rule 4) as its basis.

Either way, the disregard rule governs only execution.  A failure raised in
the parser, transform, rewrite, or planner is a separate, static layer the
rule does not touch.  And at execution there is nothing to evaluate: an
unused window is never turned into a WindowAgg node, so its DEFINE is never
reached.

So I'm convinced -- let's keep (a), the current behavior, and I'll treat
this open question as closed, with no patch change.  For the record, the two
halves stay cleanly separated: the RPR DEFINE volatility check still visits
every window clause at preprocessing (independent of select_active_windows),
while run-time DEFINE evaluation happens only for windows that survive it.

Thanks for digging up the standard text.

Best regards,
Henson

  403. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-20T03:17:19Z

    Hi Jian,

Only comments to error messages.

> CREATE TABLE stock (company TEXT, tdate DATE, price INTEGER);
> CREATE TEMP TABLE stock (company TEXT, tdate DATE, price INTEGER);
> SELECT count(*) over w FROM stock WINDOW w AS ( ROWS BETWEEN CURRENT
> ROW AND UNBOUNDED FOLLOWING PATTERN (A) DEFINE A AS
> pg_temp.stock.price > 0 );
> SELECT count(*) over w FROM stock WINDOW w AS ( ROWS BETWEEN CURRENT
> ROW AND UNBOUNDED FOLLOWING PATTERN (A) DEFINE A AS public.stock.price
>> 0 );
> SELECT count(*) over w FROM stock WINDOW w AS ( ROWS BETWEEN CURRENT
> ROW AND UNBOUNDED FOLLOWING PATTERN (A) DEFINE A AS stock.price > 0 );
> 
> The error messages for the above 3 SELECT queries are different.
> (pg_temp.stock.price, public.stock.price, stock.price) mean the same
> thing: column reference,
> Should we try to make the error messages consistent?

I have tested above queries to see how error messages actually look
like. These errors raised by different reasons and becomes different
looks natural. I see no consistency problem here.

SELECT count(*) over w FROM stock WINDOW w AS ( ROWS BETWEEN CURRENT
ROW AND UNBOUNDED FOLLOWING PATTERN (A) DEFINE A AS
pg_temp.stock.price > 0 );
(1)
psql:rangevar.sql:6: ERROR:  42601: qualified expression "pg_temp.stock.price" is not allowed in DEFINE clause
LINE 3: pg_temp.stock.price > 0 );
        ^
LOCATION:  transformColumnRef, parse_expr.c:966

"stock" table in the FROM clause is actually pg_temp.stock. The
expression "pg_temp.stock.price > 0" is valid in general but in a DEFINE
clause schema qualified column reference is not allowed by the
standard. So the error messages look reasonable to me.

SELECT count(*) over w FROM stock WINDOW w AS ( ROWS BETWEEN CURRENT
ROW AND UNBOUNDED FOLLOWING PATTERN (A) DEFINE A AS public.stock.price
> 0 );
(2)
psql:rangevar.sql:9: ERROR:  42P01: invalid reference to FROM-clause entry for table "stock"
LINE 2: ...W AND UNBOUNDED FOLLOWING PATTERN (A) DEFINE A AS public.sto...
                                                             ^
DETAIL:  There is an entry for table "stock", but it cannot be referenced from this part of the query.
LOCATION:  errorMissingRTE, parse_relation.c:3864

"stock" table in the FROM clause is actually pg_temp.stock. The
expression "public.stock.price > 0" is not valid because public.stock
is not in the FROM clause. The error messages look reasonable to me.

SELECT count(*) over w FROM stock WINDOW w AS ( ROWS BETWEEN CURRENT
ROW AND UNBOUNDED FOLLOWING PATTERN (A) DEFINE A AS stock.price > 0 );
(3)
psql:rangevar.sql:11: ERROR:  42601: range variable qualified expression "stock.price" is not allowed in DEFINE clause
LINE 2: ...W AND UNBOUNDED FOLLOWING PATTERN (A) DEFINE A AS stock.pric...
                                                             ^
LOCATION:  transformColumnRef, parse_expr.c:674

The error message precisely points out that the range variable "stock"
qualifies "stock.price", which is not allowed by the standard. I see
no problem here.

> ERROR:  range variable qualified expression "rpr_composite.items" is
> not allowed in DEFINE clause
> 
> "Range variable qualified expression" is non-standard that may confuse users.

Which part of it do you think "non-standard"?  The standard uses both
terms "Range variable" and "qualified".

> To improve clarity and consistency, let's align this with the
> established error pattern:
> 
> ERROR: invalid reference to FROM-clause entry for table "the_table"

-1. As I explained above, these 3 errors raised by the different
 reasons. "invalid reference to FROM-clause entry for table
 "the_table" is only applied to (2). So unified (1) and (3) will make
 more confusion.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  404. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-20T09:58:25Z

    Hi Henson,

> rest.  So evaluating its DEFINE -- option (b) -- would mean running a clause
> the standard tells us to ignore.

Right.

> Agreed, and the parallel is convincing.  An unused window already escapes
> the negative-frame-offset check, so singling out DEFINE as the one clause
> that must fire regardless would be inconsistent with how the rest of an
> unused window already behaves.
> 
> This also matches the planner: select_active_windows() drops any window with
> no referencing WindowFunc, and its comment already cites the same <window
> clause> General Rules (General Rule 4) as its basis.
> 
> Either way, the disregard rule governs only execution.  A failure raised in
> the parser, transform, rewrite, or planner is a separate, static layer the
> rule does not touch.  And at execution there is nothing to evaluate: an
> unused window is never turned into a WindowAgg node, so its DEFINE is never
> reached.

Yes, the rule only applies to execution. I think what we do in other
phases is implementation dependent.

> So I'm convinced -- let's keep (a), the current behavior, and I'll treat
> this open question as closed, with no patch change.  For the record, the two
> halves stay cleanly separated: the RPR DEFINE volatility check still visits
> every window clause at preprocessing (independent of select_active_windows),
> while run-time DEFINE evaluation happens only for windows that survive it.
> 
> Thanks for digging up the standard text.

You are welcome.  Please feel free to ask me if you want to know what
9075-2 says about someting.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  405. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-21T06:39:11Z

    Hi hackers,

This is an increment on top of v49: it lands the two fixes I left as
still-to-come there -- the DEFINE-evaluation use-after-free, now a dedicated
ExprContext, and the PREV/NEXT/FIRST/LAST namespace collision -- adds a
correctness fix found while reviewing the tuplestore spool (dormant
matches), and applies Jian He's and Tatsuo Ishii's review of the v48 series
as a set of mostly behavior-neutral commits.

Before the patch list, a note on CI: cfbot has been red here, but the
failure is not RPR -- it is the libLLVM 19 + ASAN JIT crash (CF 6870), which
reproduces on plain master.  The build-system fix (exclude sanitizer flags
from JIT bitcode generation) is Matheus Alcantara's; the meson half is
v3-0001-Exclude-sanitizer-flags-from-LLVM-JIT-bitcode-gen.patch:


https://postgr.es/m/CAAAe_zBX5uV9K0ikuROLgdNvDCgGqHRskT-73L+oX9=3aXR2AQ@mail.gmail.com

For v50, what would you think about folding that meson patch in as a
temporary prerequisite at the front of the series, so cfbot's ASAN build
gets past the JIT crash and actually exercises RPR?

First, two cleanups splitting changes unrelated to RPR out of the feature
patch:

  nocfbot-0001  Remove blank-line changes unrelated to row pattern
                recognition
  nocfbot-0002  Remove unnecessary includes from the row pattern
                recognition patch

The fixes -- the two I left as still-to-come in v48, plus the dormant-match
fix found since (nocfbot-0003..0005, all behavior-changing):

  nocfbot-0003  Recognize row pattern navigation operations by name in
                DEFINE
      The placeholder PREV/NEXT/FIRST/LAST pg_proc functions polluted the
      ordinary function namespace and could be silently misbound to
      same-named user functions.  They are dropped; an unqualified
      PREV/NEXT/FIRST/LAST inside a DEFINE clause is recognized as
      navigation by name, while a schema-qualified call still reaches an
      ordinary function.  A navigation operation inside a navigation offset
      -- which must be a run-time constant -- is now rejected instead of
      crashing the planner.  (f).prev follows attribute notation as an
      ordinary function, the same inside and outside DEFINE.

  nocfbot-0004  Use a dedicated ExprContext for RPR DEFINE clause
                evaluation
      nocfbot-0039's interim leak fix had turned into a use-after-free and
      was voided in v48.  DEFINE evaluation now has its own ExprContext,
      reset once per row and separate from both the per-output-tuple context
      and tmpcontext, resolving the leak and the use-after-free together.

  nocfbot-0005  Drive RPR row pattern matching once per row
      Matching only advanced when a window function read the frame, so a row
      whose only window function skips the frame (e.g. nth_value() with a
      NULL offset) left the match behind the current row -- silently wrong
      results and a spurious "cannot fetch row ... before WindowObject's
mark
      position" error.
      The match is now driven once per row, before the window functions run;
      the navigation mark advances from the frontier the match reached, so
      the tuplestore is trimmed sooner.

Review of v48 (Jian He, and Tatsuo Ishii), as nocfbot-0006..0013 --
behavior-neutral except where tagged [behavior change]:

  nocfbot-0006  Tidy up row pattern recognition plumbing
      Remove the dead collectPatternVariables()/buildDefineVariableList()
      helpers; drop the redundant rpSkipTo/defineClause arguments of
      make_windowagg(); mark RPRNavExpr.resulttype query_jumble_ignore;
      rename WindowAggState.defineClauseList to defineClauseExprs; assorted
      block flattening.  No change to planner or executor output.

  nocfbot-0007  Further tidy up row pattern recognition plumbing
      Drop the now-unused WindowClause argument of transformDefineClause();
      use foreach_node()/foreach_current_index() in the DEFINE walkers and
      drop their redundant end-of-list break tests; minor include and
      comment fixups.  No output change.

  nocfbot-0008  Refactor transformDefineClause in row pattern recognition
                [behavior change]
      Hoist the "DEFINE variable not used in PATTERN" cross-check out of the
      recursive walker into its caller, and reorder per-variable processing
      to transformExpr -> coerce_to_boolean -> pull_var_clause, dropping the
      separate second coercion pass.  The only observable change is one
      error-cursor position: the duplicate-variable error now points at the
      later definition.  New regression coverage for DEFINE coercion and Var
      propagation is added.

  nocfbot-0009  Replace a bare block with an else in the RPR DEFINE clause
                walker
      Cosmetic flattening of define_walker()'s phase dispatch into an
      if / else if / else chain.

  nocfbot-0010  Rename loop index variables in row pattern deparse helpers
      Tatsuo's suggestion; descriptive names for the deparse index/loop
      variables, no change to deparsed output.

  nocfbot-0011  Rename absorption "judgment point" to "comparison point" in
                comments
      Comment and executor-README wording only; identifiers unchanged.

  nocfbot-0012  Improve comments, documentation, and naming for row pattern
                recognition
      A batch of comment/doc clarity fixes -- the AST-vs-parse-tree wording,
      the Run Condition EXPLAIN test, the contain_rpr_walker comment, the
      ALT-marker and quantifier comments, the RPCommonSyntax.location "or
-1"
      convention, and the transformDefineClause header -- plus renaming the
      saturated-count sentinel RPR_COUNT_MAX to RPR_COUNT_INF for
      consistency with RPR_QUANTITY_INF.

  nocfbot-0013  Document eval_nav_offset_helper's NULL/negative offset
                handling
      Comment only.  The NEEDS_EVAL offset branch is reachable (a Param
      offset can be NULL or negative at run time), so it stays a graceful
      return rather than an assertion.


For traceability, where each patch came from:

  patch          proposer        proposal
  -------------  --------------
 -------------------------------------------------
  nocfbot-0001   Henson          drop blank-line churn unrelated to RPR
  nocfbot-0002   Henson          drop unnecessary includes
  nocfbot-0003   Henson          nav namespace collision; (f).prev as
ordinary function
  nocfbot-0004   Henson          dedicated ExprContext for DEFINE evaluation
  nocfbot-0005   Henson          drive row pattern matching once per row
  nocfbot-0006   Jian He         tidy RPR plumbing
  nocfbot-0007   Henson          further plumbing tidy
  nocfbot-0008   Jian He         refactor transformDefineClause
  nocfbot-0009   Jian He         bare block -> else in the DEFINE walker
  nocfbot-0010   Tatsuo Ishii    rename deparse loop/index variables
  nocfbot-0011   Jian He         "judgment point" -> "comparison point"
wording
  nocfbot-0012   Jian He         comment/doc clarity batch + RPR_COUNT_INF
rename
  nocfbot-0013   Jian He         document eval_nav_offset NULL/negative
offset handling


Still to come -- Jian He's follow-ups on this thread (2026-06-19), to fold
once each approach is agreed (all under review):

  topic                          proposed change
  -----------------------------  -------------------------------------------
  DEFINE qualified column-ref    make the differing pg_temp.t.c /
   error messages                 public.t.c / t.c messages consistent, on
                                  the standard "invalid reference to
                                  FROM-clause entry for table"
  validateRPRPatternVarCount     rename to preprocessRPRPattern
  quantifier INF bound           abstract behind an RPR_QUANTITY_INF macro
  "nullable" variables           rename to match_empty
  splitRPRTrailingAlt            use foreach_node / foreach_current_index
  buildRPRPattern signature      pass the WindowClause directly
  collectDefineVariables /       inline and drop the helpers
   tryUnwrapSingleChild
  variable-limit error msg       fold the maximum into the primary message


Quality work to run alongside, on Linux: Valgrind (leak and use-after-free,
to exercise the DEFINE ExprContext fix, whose reproducer is cassert-only)
and standing gcov coverage to catch untested planner paths.


Longer term, and out of scope for this CF entry:

  - Smaller documentation and error-message follow-ups (glossary, the
    bounded-quantifier message, README wording).
  - Short-circuit / tri-state DEFINE evaluation, as a separate series.
  - SEEK clause support (SQL:2016).
  - Empty pattern PATTERN () -- correctly rejected today; deferred because
    its empty-match semantics (SHOW/OMIT EMPTY MATCHES) are tied to the
    still-out-of-scope MEASURES clause.
  - Relaxing the DEFINE-subquery over-rejection where the standard permits
    it (the subquery does no RPR of its own and makes no outer reference).
  - Prefix-pattern absorption (an optimization; designed and intentionally
    split out as its own series).
  - R010 (MATCH_RECOGNIZE in the FROM clause) and the shared RPRContext it
    would back.


Please let me know if any of the slicing or grouping looks off.

On a personal note, some fatigue has built up, so I'll be easing off the
pace a little for a while and may be slower to follow up on this thread
than I have been.  The work continues -- just at a gentler pace.

Thanks again to Jian and Tatsuo for the careful review.

Best regards,
Henson

  406. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-21T06:52:20Z

    > CREATE TABLE stock (company TEXT, tdate DATE, price INTEGER);
> CREATE TEMP TABLE stock (company TEXT, tdate DATE, price INTEGER);
> SELECT count(*) over w FROM stock WINDOW w AS ( ROWS BETWEEN CURRENT
> ROW AND UNBOUNDED FOLLOWING PATTERN (A) DEFINE A AS
> pg_temp.stock.price > 0 );
> SELECT count(*) over w FROM stock WINDOW w AS ( ROWS BETWEEN CURRENT
> ROW AND UNBOUNDED FOLLOWING PATTERN (A) DEFINE A AS public.stock.price
>> 0 );
> SELECT count(*) over w FROM stock WINDOW w AS ( ROWS BETWEEN CURRENT
> ROW AND UNBOUNDED FOLLOWING PATTERN (A) DEFINE A AS stock.price > 0 );
>
> The error messages for the above 3 SELECT queries are different.
> (pg_temp.stock.price, public.stock.price, stock.price) mean the same
> thing: column reference,
> Should we try to make the error messages consistent?

Tatsuo has already covered this one, and I'm with him: I'd keep the three
as they are.  They come from three different paths -- a schema-qualified
reference rejected inside DEFINE, a public.stock that simply isn't the
FROM-clause relation, and a range-variable qualification -- so a single
message can't describe all three accurately.

It's worth saying why beyond "different paths".  In the standard a row
pattern variable is itself a range variable -- ISO/IEC 19075-5 4.10 has them
"used to qualify column references" in MEASURES and DEFINE -- so a qualified
name
such as A.price denotes the price column at the rows mapped to pattern
variable A.  That notation collides directly with table/range-variable
qualification: A.price and stock.price are spelled identically, and a
one-component qualifier is ambiguous between a pattern variable and a
relation.  Sorting out that ambiguity is precisely what separates the three
paths above, which is the other reason I'd keep them distinct rather than
fold them into one message now.

We don't implement pattern-variable-qualified references yet -- today an
unqualified column is implicitly the universal row pattern variable, and
that is all DEFINE and MEASURES accept.  I'd like to design and build that
qualifier support once this commit lands; since it sits right on top of the
name resolution you're raising here, I'd genuinely welcome your help with
both the design and the implementation, if you're interested.

> "Range variable qualified expression" is non-standard that may confuse
users.
> To improve clarity and consistency, let's align this with the
> established error pattern:
>
> ERROR: invalid reference to FROM-clause entry for table "the_table"

"invalid reference to FROM-clause entry for table" only fits the middle
case (public.stock); routing (1) and (3) through it would misdescribe them.
On the wording, "range variable qualified" is not really non-standard -- it
is the standard's own framing, the same one above: a row pattern variable
is a range variable, and DEFINE qualification is exactly where that
distinction becomes load-bearing once qualifier resolution lands.  A generic
FROM-clause message would erase the very difference we will need to surface
then, so I'd keep the dedicated wording rather than fold it in.

> What do you all think about renaming validateRPRPatternVarCount to
> preprocessRPRPattern?

Agreed the name is too narrow -- besides checking the count against
RPR_VARID_MAX, the function collects the unique PATTERN variable names into
p_rpr_pattern_vars (the list transformColumnRef uses to spot a
pattern-variable qualifier, and for now to reject A.price).
 preprocessRPRPattern
feels too broad, though.  Rather than pin a name down now, I'd leave some
room: the function's role will shift as the qualifier work lands, so I'll
work out a fitting name as I write that patch.

> The attached v48-0001 mainly replaces PG_INT32_MAX with
> RPR_QUANTITY_INF, but it also includes other changes.
> See the commit message for detail.

Behavior-neutral and a clear improvement; I'll take it.  I'll go through the
details closely and fold in any fixups as I write the patch.

> The attached v48-0002 patch is more about miscellaneous refactoring.
> [...]
> Refactor buildRPRPattern to accept a WindowClause pointer directly.
> [...]
> collectDefineVariables is not needed [...] Also removed
tryUnwrapSingleChild
> [...]
> Slightly adjust variable limits error message.

Also good.  The helper inlining and the WindowClause signature both read as
clear wins; I'll sort out the variable-limit message wording and the rest of
the small details as I fold it.

Thanks again for the careful work.

Best regards,
Henson

  407. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-22T07:07:43Z

    Hi Tatsuo, Jian,

Please find attached (coverage.tgz) the code coverage analysis for the RPR
branch.

* Measurement setup
- Target: PostgreSQL RPR branch, modified-lines basis (RPR-base..RPR diff)
- Build: gcc --enable-coverage with --with-llvm (LLVM JIT module included;
both C and C++ instrumented)
- Tests: make check-world (regression + TAP + contrib; no forced JIT
settings)

* Results
- Modified-line coverage: 2,608 / 2,702 (96.5%)
- Functions: 172 / 173 (99.4%)
- Breakdown of the 94 uncovered lines:
  - Reachable (coverable by tests): 28 lines
  - Unreachable (defensive / dead code): 66 lines
    - worth cleaning up (Assert / remove / coverage-exclude): 22 lines
    - best kept as idiomatic guards (enum default, pg_unreachable(),
      public windowapi.h relpos guards): 44 lines

* Projected coverage (modified-lines basis)
- Current:                                              96.5%  (2,608 /
2,702)
- After adding the proposed tests (+28 reachable):      97.6%  (2,636 /
2,702)
- After also cleaning up only the lines worth changing
  (-22; the 44 idiomatic guards are kept on purpose):   98.4%  (2,636 /
2,680)
- If every unreachable line were removed (not advised
  for the idiomatic guards):                            ~100%  (no residual
uncovered lines)

* Picking a target -- your input would help
The analysis points to three possible levels:
  (a) tests only             -> 97.6%   (add tests for the 28 reachable
lines)
  (b) tests + safe cleanups  -> 98.4%   (also Assert/remove the 22
worth-fixing lines;
                                         keep the 44 idiomatic guards as-is)
  (c) full cleanup           -> ~100%   (also rework the idiomatic guards
-- usually undesirable)
My own inclination sits somewhere between (a) and (b): add all the
reachable-line tests, and
clean up only the most clear-cut lines (e.g. the arithmetic-underflow
Assert conversions and
the dead _equalRPRPattern body), while deciding the remaining defensive
lines case by case
rather than touching all 22 at once. I'd value your view on how far to take
this.

One caveat: this reachable/unreachable classification was produced with AI
assistance, so it
may be wrong in places. When I actually write the test cases I will
scrutinize each item
individually and verify it against a coverage build before proposing
anything.

* What the report contains
Each uncovered line carries a collapsible box stating:
- Reachable / Unreachable classification + confidence
- Reachable: the concrete SQL test that covers it
- Unreachable: the reason it cannot execute, and the recommended source
change
Consecutive lines on the same straight-line flow (no branch in between) are
merged into one box.

* How to view
  tar xzf coverage.tgz
  # open coverage/index.html in a browser
  #  -> pick a file -> expand the collapsible box under each red
(uncovered) line

* Contents
- coverage/index.html  : per-file coverage overview
- coverage/html/       : per-file detail (source + uncovered-line analysis)
- coverage/untested.md : checklist of uncovered lines

Please review. Feel free to reply with any questions.

Best regards,
Henson

  408. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-22T12:55:36Z

    Hi Henson,

> Before the patch list, a note on CI: cfbot has been red here, but the
> failure is not RPR -- it is the libLLVM 19 + ASAN JIT crash (CF 6870), which
> reproduces on plain master.  The build-system fix (exclude sanitizer flags
> from JIT bitcode generation) is Matheus Alcantara's; the meson half is
> v3-0001-Exclude-sanitizer-flags-from-LLVM-JIT-bitcode-gen.patch:
> 
> 
> https://postgr.es/m/CAAAe_zBX5uV9K0ikuROLgdNvDCgGqHRskT-73L+oX9=3aXR2AQ@mail.gmail.com
> 
> For v50, what would you think about folding that meson patch in as a
> temporary prerequisite at the front of the series, so cfbot's ASAN build
> gets past the JIT crash and actually exercises RPR?

I have sent an email to those who are working on CI to let them know
the CI is partly broken and the fix has been proposed.
https://www.postgresql.org/message-id/20260622.192319.148395607405547262.ishii%40postgresql.org

Once the fix is installed, the patch does not need to be part of
v50. Let's wait for a while.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  409. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-23T14:27:15Z

    Hi hackers,

This is v50, folding in the outstanding review.  From my side there are no
known issues left, so I think it makes a good starting point for the kind of
thorough public review the feature should get before commit.

On scope: the feature is almost entirely additive.  It adds no SQL-visible
objects -- no DDL, catalog entries, or built-in functions -- and no on-disk
or catalog-format change, so it needs no catversion bump.  Row pattern
recognition is reached only through the new window-specification grammar and
the window-function machinery it extends; code that does not use it follows
the same paths as before.  The footprint is narrow by design, so the risk to
existing behaviour stays low.  It is not meant for back-porting, but it is
contained enough that a back-port would be feasible -- a measure of how
little it disturbs anything else.

That said, the feature is not simple inside, and review would pay off most
on
the parts that took the most care: the NFA matcher (alternation, reluctant
quantifiers, nullable / empty-match handling, and the dormant matches in the
tuplestore spool); the executor's tuple-slot navigation (the
EEOP_RPR_NAV_SET
/ _RESTORE steps and the slot caching behind PREV / NEXT / FIRST / LAST);
and
the planner/executor optimizations -- context absorption and the
reduced-frame
navigation that lets the tuplestore be trimmed early.  The surrounding
plumbing is, by comparison, mechanical.

The feature is unchanged since v49 and is re-attached as v49-0001..0009 so
the series applies on master; the work since v49 is the increment
v50-0001..0020 on top.

A note on the posting itself first: the v49 increment went out as
nocfbot-*.txt attachments, which cfbot does not apply, and a later
coverage-report attachment (no patches inside) left the entry with nothing
applicable -- so cfbot has been building plain master, and its current green
is just master's, with no RPR patch applied.  v50 goes out as a normal
format-patch series so cfbot picks it up and actually exercises RPR.

One ASAN task is likely to go red once it does: the libLLVM 19 + ASAN JIT
crash (CF 6870) is still unfixed in master, and RPR exercises the JIT paths,
so the ASAN build will hit it -- a toolchain/master issue, not RPR.  Matheus
Alcantara's meson fix (exclude sanitizer flags from JIT bitcode generation)
addresses it but is not merged yet; it is not folded into this series.

First, two cleanups splitting changes unrelated to RPR out of the feature
patch:

  v50-0001  Remove blank-line changes unrelated to row pattern recognition
  v50-0002  Remove unnecessary includes from the row pattern recognition
patch

The fixes (v50-0003..0005, all behavior-changing):

  v50-0003  Recognize row pattern navigation operations by name in DEFINE
  v50-0004  Use a dedicated ExprContext for RPR DEFINE clause evaluation
  v50-0005  Drive RPR row pattern matching once per row

Review of v48 (Jian He, and Tatsuo Ishii), v50-0006..0013 --
behavior-neutral
except where tagged [behavior change]:

  v50-0006  Tidy up row pattern recognition plumbing
  v50-0007  Further tidy up row pattern recognition plumbing
  v50-0008  Refactor transformDefineClause in row pattern recognition
            [behavior change]
  v50-0009  Replace a bare block with an else in the RPR DEFINE clause
walker
  v50-0010  Rename loop index variables in row pattern deparse helpers
  v50-0011  Rename absorption "judgment point" to "comparison point" in
comments
  v50-0012  Improve comments, documentation, and naming for row pattern
recognition
  v50-0013  Document eval_nav_offset_helper's NULL/negative offset handling

New since v49 -- the agreed Jian He follow-ups and the standing quality work
(v50-0014..0020):

  v50-0014  Tidy up the row pattern unbounded-quantifier sentinel
  v50-0015  Simplify row pattern compilation by passing the WindowClause
  v50-0016  Reword the row pattern variable-limit error
  v50-0017  Reformat row pattern regression tests for readability
  v50-0018  Add row pattern recognition coverage tests and tidy unreachable
code
  v50-0019  Free RPR NFA states with pfree() under USE_VALGRIND
  v50-0020  Clarify row pattern recognition comments on "step" and no_equal

For traceability, where each patch came from:

  patch     proposer       proposal
  --------  -------------
 --------------------------------------------------
  v50-0001  Henson         drop blank-line churn unrelated to RPR
  v50-0002  Henson         drop unnecessary includes
  v50-0003  Henson         nav namespace collision; (f).prev as ordinary
function
  v50-0004  Henson         dedicated ExprContext for DEFINE evaluation
  v50-0005  Henson         drive row pattern matching once per row
  v50-0006  Jian He        tidy RPR plumbing
  v50-0007  Henson         further plumbing tidy
  v50-0008  Jian He        refactor transformDefineClause
  v50-0009  Jian He        bare block -> else in the DEFINE walker
  v50-0010  Tatsuo Ishii   rename deparse loop/index variables
  v50-0011  Jian He        "judgment point" -> "comparison point" wording
  v50-0012  Jian He        comment/doc clarity batch + RPR_COUNT_INF rename
  v50-0013  Jian He        document eval_nav_offset NULL/negative offset
handling
  v50-0014  Jian He        abstract the quantifier INF bound / sentinel
  v50-0015  Jian He        pass the WindowClause into pattern compilation
  v50-0016  Jian He        fold the maximum into the variable-limit message
  v50-0017  Henson         reformat the regression tests
  v50-0018  Henson         coverage tests + tidy unreachable code
  v50-0019  Henson         free NFA states with pfree() under USE_VALGRIND
  v50-0020  Jian He        clarify the "step" and no_equal comments

Disposition of the latest review (Jian He -- the on-list v48 follow-ups and
the off-list thread that produced v50-0020):

  proposal                              disposition
  ------------------------------------
 -------------------------------------
  RPR_QUANTITY_INF sentinel unify       accepted -> v50-0014
  pass WindowClause; inline             accepted -> v50-0015
   collectDefineVariables /
   tryUnwrapSingleChild
  variable-limit error reword           accepted -> v50-0016 (kept errmsg +
                                         errdetail, not a single message)
  define "step"; reword the no_equal    accepted -> v50-0020 (off-list)
   comment
  "nullable" -> match_empty             rejected -- "nullable" is the
standard
                                         automata term
  unify DEFINE qualified column-ref     rejected -- keep the dedicated
wording
   error messages                        (the range-variable distinction is
                                         needed for the qualifier work)
  validateRPRPatternVarCount ->         deferred -- name to be settled with
   preprocessRPRPattern                  the qualifier patch
  (A B+)+ "variable-length element"     rejected -- the original is
accurate;
   -> "quantifier is different"          cf. (A{2} B{3})+, which is
absorbable
   (off-list)

Quality work (Linux), each sent as its own follow-up on this thread:

  - Valgrind (leak / use-after-free): no errors across parse, NFA build,
    pattern scan, EXPLAIN and node serialization; exercises the DEFINE
    ExprContext fix, whose reproducer is cassert-only.
  - gcov coverage: modified-line 98.4% (functions 100%); the remaining
    uncovered lines are idiomatic defensive guards (enum-default branches,
    pg_unreachable(), the public windowapi.h relpos guards) kept on purpose.

Both figures were measured at v50-0019; v50-0020 is comment-only, so they
carry over unchanged.  Each report goes out as a separate message so the
details and attachments stay out of this patch mail.

Longer term, and out of scope for this CF entry:

  - Smaller documentation and error-message follow-ups (glossary, the
    bounded-quantifier message, README wording).
  - Short-circuit / tri-state DEFINE evaluation, as a separate series.
  - SEEK clause support (SQL:2016).
  - Empty pattern PATTERN () -- correctly rejected today; deferred because
    its empty-match semantics (SHOW/OMIT EMPTY MATCHES) are tied to the
    still-out-of-scope MEASURES clause.
  - Relaxing the DEFINE-subquery over-rejection where the standard permits
it.
  - Prefix-pattern absorption (an optimization; split out as its own
series).
  - R010 (MATCH_RECOGNIZE in the FROM clause) and the shared RPRContext it
    would back.

Please let me know if any of the slicing or grouping looks off.

Thanks again to Jian and Tatsuo for the careful review.

Best regards,
Henson

  410. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-23T14:28:18Z

    Hi hackers,

Following up on the coverage analysis I posted earlier: I've since pushed
the
work it proposed, so please find attached (coverage.tgz) an updated report
that
reflects the committed changes. In short, the branch now lands at the
target we
discussed.

* What changed since the first report
- New commits add the row pattern recognition coverage tests for the
reachable
  lines, and tidy up the unreachable defensive/dead code (Assert
conversions,
  removing the dead _equalRPRPattern body, and similar).
- This is essentially the (b) path from the earlier mail: add the
reachable-line
  tests plus the safe cleanups, while keeping the idiomatic guards as-is.

* Measurement setup
- Target: PostgreSQL RPR branch, modified-lines basis (RPR-base..RPR diff)
- Build: gcc --enable-coverage with --with-llvm (LLVM JIT module included;
both C and C++ instrumented)
- Tests: make check-world (regression + TAP + contrib; no forced JIT
settings)

* Results (vs. the first report)
- Modified-line coverage: 96.5% (2,608 / 2,702)  ->  98.4% (2,635 / 2,679)
- Functions:              99.4% (172 / 173)      ->  100%  (170 / 170)
- Remaining 44 uncovered lines are the idiomatic defensive guards we agreed
to
  keep on purpose:
  - enum-default branches
  - pg_unreachable()
  - public windowapi.h relpos guards
  I left these untouched rather than reworking them to chase ~100%, which
  matches my earlier inclination and our reading of where the line should
sit.

* On the earlier caveat
The first report flagged that the reachable/unreachable classification was
AI-assisted and might be wrong. That is now resolved in practice: each line
I
claimed was coverable was verified against a coverage build as I wrote its
test,
rather than resting on the classification.

* What the report contains
Each uncovered line carries a collapsible box stating:
- Reachable / Unreachable classification + confidence
- Reachable: the concrete SQL test that covers it
- Unreachable: the reason it cannot execute, and the recommended source
change
Consecutive lines on the same straight-line flow (no branch in between) are
merged into one box.

* How to view
  tar xzf coverage.tgz
  # open coverage/index.html in a browser
  #  -> pick a file -> expand the collapsible box under each red
(uncovered) line

* Contents
- coverage/index.html  : per-file coverage overview
- coverage/html/       : per-file detail (source + uncovered-line analysis)
- coverage/untested.md : checklist of uncovered lines

Please review. Feel free to reply with any questions.

Best regards,
Henson

  411. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-23T14:28:39Z

    Hi hackers,

As a final memory-safety check before wrapping up the RPR patch, I ran the
branch under Valgrind. The short version: no memory errors were found.

Setup, on Linux/AMD64 (Valgrind is not usable on macOS):

  configure: --enable-debug --enable-cassert -DUSE_VALGRIND
  also defined: -DCOPY_PARSE_PLAN_TREES -DWRITE_READ_PARSE_PLAN_TREES
                -DRAW_EXPRESSION_COVERAGE_TEST
  valgrind: --suppressions=src/tools/valgrind.supp --trace-children=yes
            --track-origins=yes --leak-check=no --error-exitcode=128

The round-trip defines keep the parse/plan-tree copy and write/read paths
on by
default, so _outRPRPattern, _readRPRPattern and _copyRPRPattern are
exercised on
every plan tree, not just at parse time.

I ran installcheck against this instance in three phases.

Phase A, round-trips on: rpr, rpr_nfa, rpr_explain and rpr_integration.
These
are normal-size patterns, so the node copy/write-read paths are verified
along
the way. All passed.

Phase B, round-trips off: rpr_base, which includes the large 32767-element
pattern. Serializing a tree that big under Valgrind is impractically slow,
and
the serialization path is already covered by the normal-size patterns in
Phase A, so I disabled debug_copy_parse_plan_trees and
debug_write_read_parse_plan_trees here to exercise the NFA build and
scanRPRPattern's large arrays without the serialization bottleneck. Passed.

Phase C: window, preceded by test_setup, create_index and sanity_check so
the
table, indexes and stats are in place. This one reported a regression diff:
a
few EXPLAIN plans came out with Seq Scan / Hash Join where the expected
output
has Index Only Scan / Nested Loop or Merge Join. That is a planner artifact
of
running window outside the full parallel schedule (a different stats/cost
environment), not anything RPR changed; the same queries match expectations
under a normal check-world run. I included window mainly to exercise the
WindowAgg/RPR interaction under instrumentation, and for that the diff does
not
matter, because the memory verdict comes from the Valgrind logs rather than
the
regression output.

Results: no VALGRINDERROR markers in any per-pid log, and every log is empty
(with --quiet, a clean run leaves 0-byte logs), window included. The
postmaster
exited 0, so --error-exitcode=128 never fired, and the tests ran tens of
times
slower than usual, which confirms the code really did run under
instrumentation.
So across parse, NFA build, pattern scan, EXPLAIN and node serialization, no
invalid read/write, uninitialised-value use, out-of-bounds access or invalid
free was observed.

The usual caveat applies: this is dynamic analysis, so it only attests to
the
paths the tests actually exercised. As I add the coverage tests discussed in
the other thread, I will re-run the same setup over the expanded suite.

Best regards,
Henson

  412. Re: Row pattern recognition

    jian he <jian.universality@gmail.com> — 2026-06-24T02:33:24Z

    Hi.

The attached patch refactors absorbability helper functions, white at
it, also other misc changes.
It's still based on https://github.com/assam258-5892/postgres/commits/RPR

The below is commit message:
----------------------------------------------------------------------------
Rename isUnboundedStart() to start_with_unbounded_quantifier() and
isFixedLengthChildren() to isFixedQuantifier() for clarity.

Rework start_with_unbounded_quantifier() to accept an RPRPatternElement *
directly instead of an index, and restructure its body as an explicit
if/else chain over element kinds (VAR, BEGIN, ALT/FIN, END) rather than
two loosely-coupled early-return checks.

Fix subgroup traversal in isFixedQuantifier() to use elem->jump - 1 to
locate the matching END element directly, replacing the previous loop
that walked next pointers.

Move isAbsorbable initialization from finalizeRPRPattern() into
makeRPRPattern(), and guard the computeAbsorbabilityRecursive() call in
computeAbsorbability() so it is only entered when the first element is
an ALT or has an unbounded quantifier.

Use palloc_array/palloc0_array in makeRPRPattern(). Remove the redundant
"T_RPRPattern" comment on the NodeTag field. Update README.rpr and
in-code comments to reflect the new function names.
----------------------------------------------------------------------------



--
jian
https://www.enterprisedb.com/

  413. Re: Row pattern recognition

    jian he <jian.universality@gmail.com> — 2026-06-24T05:12:30Z

    On Wed, Jun 24, 2026 at 10:33 AM jian he <jian.universality@gmail.com> wrote:
>
> Hi.
>
> It's still based on https://github.com/assam258-5892/postgres/commits/RPR
>

    /*
     * Once the name matches we never fall back to function resolution, so any
     * decoration that does not make sense for a navigation operation is a
     * hard error.  The aggregate/window decorations (agg_star, DISTINCT,
     * WITHIN GROUP, ORDER BY, FILTER, OVER, RESPECT/IGNORE NULLS) are already
     * rejected by the common path in ParseFuncOrColumn, which treated the
     * recognized name as an ordinary function; what remains are the
     * decorations that path accepts for a plain function but a navigation
     * operation must still reject.
     */

The above comments can be deleted, ParseRPRNavCall, ParseFuncOrColumn
already have lots of comments.
Also the error check and its message are quite intuitive in ParseRPRNavCall.

coerce_to_common_type have comments:
```
 * This is used following select_common_type() to coerce the individual
 * expressions to the desired type
 ```

coerce_to_target_type comments says
```
* This is the general-purpose entry point for arbitrary type coercion
* operations.
```
So, I tended to use coerce_to_target_type.

ParseRPRNavCall ending can be simplified because coerce_to_target_type
can handle the same data type.
Therefore, `if (offtype != INT8OID)` is not necessary.

Drop the extra parentheses around ereport() argument lists, fewer
parentheses are always better for new code.



--
jian
https://www.enterprisedb.com/

  414. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-24T06:04:09Z

    Hi jian,

Thanks for the refactor.  I went through it change by change -- the
per-file notes are at the bottom.

A note up front, so the notes don't read as dismissive: at this stage I'd
rather prioritize validated code over better code.  The branch has a lot of
coverage and Valgrind behind it, and every change -- even a clean,
equivalent one -- moves a line out from under that.  So where I say "drop",
it means "not now", not "not good"; most of these would be welcome as a
follow-up.  One piece, though, does need a fix before any of it lands.

That piece is the new guard in computeAbsorbability(): it carries a
behavior change that no test covers.

> Move isAbsorbable initialization from finalizeRPRPattern() into
> makeRPRPattern(), and guard the computeAbsorbabilityRecursive() call in
> computeAbsorbability() so it is only entered when the first element is
> an ALT or has an unbounded quantifier.

First, what "absorbable" means here: the first unbounded repetition of a
fixed-length body reachable from the start of the pattern.  This rule is
worth writing down -- in README.rpr and in the comment on
computeAbsorbability() (or wherever fits best) -- so it is stated once,
explicitly (patch attached: nocfbot-absorb-definition-doc).

By that definition the leading A+ in (A+ B){2,3} is absorbable, but the
patch drops it.

The offending check is in computeAbsorbability():

    if (RPRElemIsAlt(&pattern->elements[0]) ||
        pattern->elements[0].max == RPR_QUANTITY_INF)
        computeAbsorbabilityRecursive(pattern, 0, &hasAbsorbable);

It looks only at elements[0], so a bounded outer group's BEGIN (finite max)
fails the check and the recursion that would descend to A+ never runs.

I'm attaching a small test for this (nocfbot-absorb-bounded-outer-test): it
passes on the branch and fails with v50-0001 applied, pinning both the a+"
marker and the absorbed-context count.

For the rest, change by change -- "ok" = take as-is, "drop" = leave out:

src/backend/optimizer/plan/rpr.c

The branch is already validated under coverage and Valgrind, so I'd rather
not re-open verified logic for equivalent refactors -- glad to take those
as a follow-up.  In file order:

  ok    palloc_array / palloc0_array in makeRPRPattern()
      result->varNames = palloc_array(char *, numVars);
      result->elements = palloc0_array(RPRPatternElement, numElements);

  ok    move the isAbsorbable = false init into makeRPRPattern()

  drop  rename beginElem -> elem in fillRPRPatternGroup()
        only beginElem was renamed; with endElem kept, the begin/end pair is
        no longer symmetric (elem / endElem)

  drop  rename isFixedLengthChildren -> isFixedQuantifier -- keep the old
name
        "fixed-length" is the term everywhere else (README.rpr, the nearby
        comments, even this function's own body), and it drops the
"children"
        that flags a whole-subtree check

  drop  the jump-1 traversal rewrite in that function -- equivalent
refactor,
        defer to a follow-up

  drop  rename to start_with_unbounded_quantifier -- keep isUnboundedStart
    Lone snake_case among camelCase neighbors -- including
isFixedQuantifier,
    renamed in this same patch -- and it reads as a predicate though it
sets flags.

  drop  start_with_*'s new signature + if/else restructure -- equivalent
        refactor; keep the original isUnboundedStart body

  drop  rename branchFirst -> branchElement -- cosmetic churn, no behavior
change

  drop  the new computeAbsorbability() guard -- the regression above
      if (RPRElemIsAlt(&pattern->elements[0]) ||
          pattern->elements[0].max == RPR_QUANTITY_INF)

src/include/nodes/plannodes.h

  ok    drop the redundant /* T_RPRPattern */ on the type field
      NodeTag type;

  drop  reworded "see isUnboundedStart" block comment -- keep the original
        (it still matches now that isUnboundedStart stays)

  ok    drop the inline isAbsorbable comment -- the block comment above
covers it
      bool isAbsorbable;

src/backend/executor/README.rpr

  ok    spell out the skip mode -- a clear doc fix on its own
      (1) SKIP PAST LAST ROW (not SKIP TO NEXT ROW)

  drop  isUnboundedStart -> start_with_unbounded_quantifier -- follows the
rename

Tatsuo, what's your read on all this -- in particular, hold the equivalent
refactors until after commit, or take them now?

Best regards,
Henson

  415. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-24T07:45:13Z

    Hi jian,

Thanks -- this is a clean improvement, no objections. Honestly this part of
the
parser is an area where you read it better than I do, so I'm glad to defer
to
you here. Quick notes inline.

> The above comments can be deleted, ParseRPRNavCall, ParseFuncOrColumn
> already have lots of comments.
> Also the error check and its message are quite intuitive in
ParseRPRNavCall.

Agreed.

> So, I tended to use coerce_to_target_type.

Agreed.

> ParseRPRNavCall ending can be simplified because coerce_to_target_type
> can handle the same data type.
> Therefore, `if (offtype != INT8OID)` is not necessary.

Agreed.

> Drop the extra parentheses around ereport() argument lists, fewer
> parentheses are always better for new code.

Agreed.

Tatsuo, this one looks good to take whenever you like.

Best,
Henson

  416. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-26T01:23:53Z

    Hi all,

A quick heads-up for this thread, cross-referenced from another one.

Haibo Yan has a patch series adding DISTINCT support to plain aggregate
window functions [1]. As it touches window-aggregate frame handling, I
applied it on top of the current RPR patch set and built the two together.
They combine cleanly today: no rebase conflict, and the regression tests
pass with no change to code or tests.

This is structural, not luck. RPR requires the frame to start at CURRENT
ROW, while that feature requires UNBOUNDED PRECEDING, so a single window can
satisfy at most one of them -- no overlap.

That said, the DISTINCT roadmap plans to relax its UNBOUNDED-PRECEDING
restriction (the sliding-frame patches), while RPR's CURRENT-ROW requirement
is fixed by the standard. So as those frame restrictions are lifted the two
could start to overlap, and at that point we'd likely need to review the
related frame-handling changes together. Nothing to act on now -- just
flagging it so we keep an eye on it.

Full review of the DISTINCT series is in the other thread [1].

[1]
https://www.postgresql.org/message-id/CAAAe_zCfVbNrOUF5FaE38EcwYbtXacThR9xdVe2UJJPeDSL2KA@mail.gmail.com

Best,
Henson

  417. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-26T01:37:23Z

    Hi all,

One small follow-up on top of v50.

Master recently reworded the error raised when RESPECT/IGNORE NULLS is used
on a non-window function, from

  "only window functions accept RESPECT/IGNORE NULLS"

to

  "RESPECT/IGNORE NULLS specified, but %s is not a window function".

rpr_base was written before that change, so one expected-output line now
mismatches and the test fails on current master. The attached patch updates
just that line. No code or behavior change.

I'm sending it as a separate attachment so the v50 series stays as posted;
it applies on top of v50-0020.

Best,
Henson

  418. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-26T02:32:16Z

    Hi Henson,

> Hi all,
> 
> A quick heads-up for this thread, cross-referenced from another one.
> 
> Haibo Yan has a patch series adding DISTINCT support to plain aggregate
> window functions [1]. As it touches window-aggregate frame handling, I
> applied it on top of the current RPR patch set and built the two together.
> They combine cleanly today: no rebase conflict, and the regression tests
> pass with no change to code or tests.
> 
> This is structural, not luck. RPR requires the frame to start at CURRENT
> ROW, while that feature requires UNBOUNDED PRECEDING, so a single window can
> satisfy at most one of them -- no overlap.
> 
> That said, the DISTINCT roadmap plans to relax its UNBOUNDED-PRECEDING
> restriction (the sliding-frame patches), while RPR's CURRENT-ROW requirement
> is fixed by the standard. So as those frame restrictions are lifted the two
> could start to overlap, and at that point we'd likely need to review the
> related frame-handling changes together. Nothing to act on now -- just
> flagging it so we keep an eye on it.
> 
> Full review of the DISTINCT series is in the other thread [1].
> 
> [1]
> https://www.postgresql.org/message-id/CAAAe_zCfVbNrOUF5FaE38EcwYbtXacThR9xdVe2UJJPeDSL2KA@mail.gmail.com

Thanks for letting know us!
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  419. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-26T02:35:05Z

    Hi Henson,

> Hi all,
> 
> One small follow-up on top of v50.
> 
> Master recently reworded the error raised when RESPECT/IGNORE NULLS is used
> on a non-window function, from
> 
>   "only window functions accept RESPECT/IGNORE NULLS"
> 
> to
> 
>   "RESPECT/IGNORE NULLS specified, but %s is not a window function".
> 
> rpr_base was written before that change, so one expected-output line now
> mismatches and the test fails on current master. The attached patch updates
> just that line. No code or behavior change.
> 
> I'm sending it as a separate attachment so the v50 series stays as posted;
> it applies on top of v50-0020.

Thanks for the patch. I will include it in the upcoming v50 patches.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  420. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-28T10:12:22Z

    Hi Tatsuo,

Two things before the next review round.  Both aim at the same thing:
keeping the v50 diff to what RPR genuinely changes, so the next round
reviews RPR on its own merits rather than hunks that drifted in or touch
pre-existing window code.

First, the cfbot is currently red on v49 across all platforms.  The cause
is not RPR: master recently reworded the RESPECT/IGNORE NULLS error, which
left rpr_base's expected output stale, so the regression diff fails "make
check" everywhere.  That is exactly the one-line fix you already agreed to
fold in; I have posted it here as v50-0021 so the series builds green on
its own.  This one is required regardless.

Second, while going over the v49 diff against master I found several hunks
that are either unrelated to RPR or that touch pre-existing window code.  I
split them into small patches so each can be decided on its own.

Patch list
----------
Every patch in this reply.  revert-* removes the change from RPR;
optional-* re-applies the same change to master separately.

  v50-0021       cfbot build fix (required regardless)
  revert-0001    drop unrelated include and stray blank line  (required
cleanup)
  revert-0004    restore doc paragraph line wrapping          (required
cleanup)
  revert-0002    #2 drop the mark-position diagnostic         (your call)
  optional-0001  #2 land the diagnostic on master
  revert-0003    #3 restore the funcname guard                (your call)
  optional-0002  #3 land the guard removal on master
  revert-0005    #5 drop the EXCLUDE TIES tests               (your call)
  optional-0003  #5 land the tests on master

Required cleanups
-----------------
These only remove changes that crept in but have nothing to do with RPR.
I split them out so you can fold them into the feature patch; being
required cleanups, I recommend applying them:

  revert-0001    drop an unrelated "nodes/plannodes.h" include and a stray
                 blank line in eval_windowaggregates()
  revert-0004    restore the line wrapping of two pre-existing doc
                 paragraphs (advanced.sgml, ref/select.sgml)

Your call
---------
The next three touch pre-existing window code, so I would rather you decide
where each belongs.  I prepared the building blocks so you can pick any
option:

  #2  window_gettupleslot() mark-position elog -- add row positions
      v49 extended the can't-happen error with the pos/markpos values.
        (a) keep it in RPR                 -> apply neither patch
        (b) drop it                        -> apply revert-0002
        (c) land it on master separately   -> revert-0002 + optional-0001
      A tiny additive diagnostic; I lean to (c), but (a) is fine too.

  #3  funcname guard in WinCheckAndInitializeNullTreatment()
      v49 removed the "could not get function name" guard as unreachable.
        (a) keep the removal in RPR        -> apply neither patch
        (b) restore the guard, drop it     -> apply revert-0003
        (c) remove it on master separately -> revert-0003 + optional-0002
      The branch is genuinely unreachable, but it drops a deliberate guard
      in pre-existing code, so of the three this one most warrants caution.
      Preserving the guard via (b) or (c) looks safest, but I will leave the
      decision to you.

  #5  EXCLUDE TIES window-frame test coverage
      Two nth_value/last_value cases.  The accounting they exercise already
      exists on master and the tests pass without RPR, so they are not RPR
      coverage.
        (a) keep them in RPR               -> apply neither patch
        (b) land them on master separately -> revert-0005 + optional-0003
      I lean to (b).

For my part, I will carry out the deeper review against v50 with v50-0021
and revert-0001/0004 applied, plus whichever of the above you choose, as
the minimal green baseline.

Best,
Henson

  421. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-28T10:22:46Z

    To: Tatsuo Ishii <ishii@postgresql.org>, jian.universality@gmail.com
Cc: pgsql-hackers@postgresql.org, zsolt.parragi@percona.com,
    sjjang112233@gmail.com, vik@postgresfriends.org, er@xs4all.nl,
    jacob.champion@enterprisedb.com, david.g.johnston@gmail.com,
    peter@eisentraut.org, li.evan.chao@gmail.com
Subject: Re: Row pattern recognition
In-Reply-To: <CAAAe_zCd+gzpYSsC04NsKyCJeneChVZCaBP540d0p2=
c3abvvw@mail.gmail.com>

Hi Tatsuo,

> v50-0021-adjust-rpr_base-expected-output.patch
> revert-0001-remove-incidental-changes.patch
> revert-0002-revert-mark-position-elog.patch
> revert-0003-restore-funcname-guard.patch
> revert-0004-restore-doc-line-wrapping.patch
> revert-0005-remove-exclude-ties-tests.patch
> optional-0001-include-row-positions-in-mark-position-error.patch
> optional-0002-remove-unreachable-funcname-guard.patch
> optional-0003-add-exclude-ties-test-coverage.patch

Apologies for the noise -- the nine attachments above are not a new full
patch series.  They are the small auxiliary patches described in that mail
(the v50-0021 build fix plus the revert-*/optional-* hunks), meant to be
applied selectively on top of the current v50 series, not to replace it.

Because I attached them as .patch files, cfbot will treat them as the latest
patch set and try to apply them to master.  They are not the series and will
fail there, so the CF entry will go red through no fault of the actual patch
-- please disregard that run.

Re-posting the full series is what restores cfbot, but a plain re-post of
v49 stays red on its own: it needs v50-0021 (the rpr_base expected-output
fix) folded in to build green.  Could you re-post the series with v50-0021
applied, so cfbot tracks the right patch set and goes green again?

Sorry again for the confusion.

Best,
Henson

  422. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-06-29T09:37:30Z

    > Hi hackers,
> 
> This is v50, folding in the outstanding review.  From my side there are no
> known issues left, so I think it makes a good starting point for the kind of
> thorough public review the feature should get before commit.

Thanks for the incremental patches. I am going to publish v50 patch series.
Here are some commnets to the v50-* patches.

> First, two cleanups splitting changes unrelated to RPR out of the feature
> patch:
>
> v50-0001  Remove blank-line changes unrelated to row pattern recognition
> v50-0002  Remove unnecessary includes from the row pattern recognition
> patch

0001 and 0002 look good to me.

> The fixes (v50-0003..0005, all behavior-changing):
> 
>   v50-0003  Recognize row pattern navigation operations by name in DEFINE

diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c
index 1f6c8fa4fb2..f3b37aa992c 100644
--- a/src/backend/parser/parse_func.c
+++ b/src/backend/parser/parse_func.c

@@ -2111,6 +2080,151 @@ FuncNameAsType(List *funcname)

+	if (fn->func_variadic)
+		ereport(ERROR,
+				(errcode(ERRCODE_SYNTAX_ERROR),
+				 errmsg("cannot use VARIADIC with row pattern navigation function %s",
+						navname),
+				 parser_errposition(pstate, location)));
+	if (argnames != NIL)
+		ereport(ERROR,
+				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+				 errmsg("row pattern navigation operations cannot use named arguments"),
+				 parser_errposition(pstate, location)));

Shouldn't the error code for the named arguments case be
ERRCODE_SYNTAX_ERROR? As far as I know the SQL standard does not allow
to use named arguments with row pattern navigation operations. So it's
not a required feature.

Also errmsg is different from surroundings. Probably "cannot use named
arguments with row pattern navigation function %s" is better?

diff --git a/src/backend/parser/parse_rpr.c b/src/backend/parser/parse_rpr.c
index 3eaea2be750..8ed01bb8f28 100644
--- a/src/backend/parser/parse_rpr.c
+++ b/src/backend/parser/parse_rpr.c
@@ -472,6 +472,7 @@ transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef,
  *     * PREV/NEXT wrapping FIRST/LAST flattens to a compound kind
  *     * Other nestings are rejected (FIRST(PREV()), PREV(PREV()), ...)
  *     * offset_arg / compound_offset_arg must not contain column refs
+ *       or nested navigation operations

Needs '*' in front of "or nested....".

>   v50-0004  Use a dedicated ExprContext for RPR DEFINE clause evaluation

Looks good to me.

Will continue reviewes tomorrow.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  423. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-06-30T00:07:11Z

    Hi Tatsuo,

Thank you very much for taking the time to start the review.  I will reply
in step, one patch at a time, from 0001 up to wherever your review has
reached -- so far that is 0001..0004 -- and I will keep following along as
you continue.

For convenience I have attached just the four patches you have reviewed,
keeping their original v50 names but renamed to nocfbot-* so that cfbot does
not pick them up:

  nocfbot-0001  Remove blank-line changes unrelated to row pattern
recognition
  nocfbot-0002  Remove unnecessary includes from the row pattern
recognition patch
  nocfbot-0003  Recognize row pattern navigation operations by name in
DEFINE
  nocfbot-0004  Use a dedicated ExprContext for RPR DEFINE clause evaluation

0001, 0002 and 0004 are unchanged from what you reviewed; I include them
only so this reply stands on its own for the reviewed range.  0003 folds in
the three points you raised below.

> 0001 and 0002 look good to me.

Thank you for confirming -- I have left them as they are.

>>   v50-0003  Recognize row pattern navigation operations by name in DEFINE
>
> + if (argnames != NIL)
> + ereport(ERROR,
> + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
> + errmsg("row pattern navigation operations cannot use named arguments"),
> + parser_errposition(pstate, location)));
>
> Shouldn't the error code for the named arguments case be
> ERRCODE_SYNTAX_ERROR? As far as I know the SQL standard does not allow
> to use named arguments with row pattern navigation operations. So it's
> not a required feature.

You are right, thank you.  The standard does not provide for named arguments
here at all, so this is a syntax violation rather than an unimplemented
feature.  I have changed the errcode to ERRCODE_SYNTAX_ERROR, which also
makes it consistent with the VARIADIC and argument-count checks alongside
it.

> Also errmsg is different from surroundings. Probably "cannot use named
> arguments with row pattern navigation function %s" is better?

Thank you for the suggested wording; I have adopted it as is:

    errmsg("cannot use named arguments with row pattern navigation function
%s",
           navname)

It now reads in the same "cannot use X with row pattern navigation
function %s" shape as the VARIADIC message just above it.  I have also
updated the expected output in rpr.sql to match (ERROR:  cannot use named
arguments with row pattern navigation function PREV).

> + *     * offset_arg / compound_offset_arg must not contain column refs
> + *       or nested navigation operations
>
> Needs '*' in front of "or nested....".

Thank you for catching this.  The last line is a continuation, but the '*'
bullets ran together with the comment's own '*' and hid that.  I switched
the bullets to '-' to match the sibling rule list in this file, so the
continuation now reads clearly:

     * ... enforces, for each outer RPRNavExpr (per ISO/IEC 19075-5 5.6.4
     * nesting rules):
     *   - arg must contain at least one column reference
     *   - PREV/NEXT wrapping FIRST/LAST flattens to a compound kind
     *   - Other nestings are rejected (FIRST(PREV()), PREV(PREV()), ...)
     *   - offset_arg / compound_offset_arg must not contain column refs
     *     or nested navigation operations

I hope this addresses what you had in mind; please let me know if you would
prefer a different form.

>>   v50-0004  Use a dedicated ExprContext for RPR DEFINE clause evaluation
>
> Looks good to me.

Thank you for confirming -- left as is.

> Will continue reviewes tomorrow.

Thank you, I look forward to it.  I will respond to each further patch as
your review reaches it, in the same one-at-a-time fashion.

Best regards,
Henson

  424. Re: Row pattern recognition

    jian he <jian.universality@gmail.com> — 2026-06-30T03:51:44Z

    Hi.

This thread has high traffic, so my review is still based on
https://github.com/assam258-5892/postgres/commits/RPR

/*
 * nfa_eval_var_match
 *
 * Evaluate if a VAR element matches the current row.
 *
 * varMatched is a pre-evaluated boolean array indexed by varId, computed
 * once per row by evaluating all DEFINE expressions.  NULL means no DEFINE
 * clauses exist (only possible during early development/testing).
 *
 * Per ISO/IEC 19075-5 Feature R020, pattern variables not listed in DEFINE
 * are implicitly TRUE -- they match every row.  This is checked via
 * varId >= list_length.
 */

``(only possible during early development/testing).`` comment is not necessary?

nfa_match desperately needs an extra i(nt64 currentPos) argument. The
whole thing execRPR.c is
fairly complicated, and adding currentPos makes it much easier to understand the
current execution state when debugging with GDB.


    /*
     * Evaluate VAR elements against current row. For VARs that reach max
     * count with END next, advance through the chain of END elements inline
     * so absorb phase can compare states at comparison points.
     */
The comments above in nfa_match are unnecessary because in ``if
(RPRElemIsAbsorbableBranch(elem)``
we already have many comments, and the meaning seems the same.

Similar to REG_DEBUG, we really need an RPR_DEBUG option. For example, in
nfa_match, it would be useful to verify that ctx->states is not a circular
linked list. When RPR_DEBUG is enabled, we can perform these kinds of sanity
checks and other debug-only validations without affecting release builds.

   <para>
    The SQL standard defines more subclauses: <literal>MEASURES</literal>
    and <literal>SUBSET</literal>. They are not currently supported
    in <productname>PostgreSQL</productname>. Also in the standard there are
    more variations in <literal>AFTER MATCH</literal> clause.
   </para>
This part should stay in the Compatibility section.



--
jian
https://www.enterprisedb.com/

  425. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-07-01T12:18:05Z

    > Will continue reviewes tomorrow.

Sorry for delay. Here is the review for
v50-0005-match-once-per-row.patch.

--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -239,6 +239,10 @@ static int64 row_is_in_reduced_frame(WindowObject winobj, int64 pos);
 
 static void clear_reduced_frame(WindowAggState *winstate);
 static int	get_reduced_frame_status(WindowAggState *winstate, int64 pos);
+static void advance_nav_mark(WindowAggState *winstate, int64 currentPos);
+static void advance_reduced_frame_nfa(WindowObject winobj,
+									  RPRNFAContext *targetCtx, int64 pos,
+									  bool hasLimitedFrame, int64 frameOffset);
 static void update_reduced_frame(WindowObject winobj, int64 pos);
 
 /* Forward declarations - NFA row evaluation */
@@ -2521,6 +2525,16 @@ ExecWindowAgg(PlanState *pstate)
 			{
 				if (winstate->rpSkipTo == ST_NEXT_ROW)
 					clear_reduced_frame(winstate);
+
+				/*
+				 * Drive the row pattern match every row, so it tracks the row
+				 * scan rather than frame access: a window function that skips
+				 * the frame (e.g. nth_value() with a NULL offset) must not
+				 * leave the match state behind currentpos.
+				 */
+				Assert(winstate->nav_winobj != NULL);
+				(void) row_is_in_reduced_frame(winstate->nav_winobj,
+											   winstate->currentpos);

Is there any reason that we call row_is_in_reduced_frame() here,
instead of something like:

   update_frameheadpos(winstate);
   update_reduced_frame(winobj, pos);

row_is_in_reduced_frame() returns row status. Ignoring it using (void)
looks a little bit confusing to me.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  426. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-07-02T00:23:28Z

    Hi Jian,

Thanks for the review, and for basing it on the RPR branch. Responses
inline.

> ``(only possible during early development/testing).`` comment is not
necessary?

Agreed, and it is actually misleading: varMatched == NULL is a live
signal, not a dev-only artifact. nfa_match() is called with NULL to force
every VAR to not-match at a frame boundary (execRPR.c, "Frame boundary
reached: force mismatch"). I will rewrite the comment to describe that
meaning instead of the "early development/testing" note.

> nfa_match desperately needs an extra i(nt64 currentPos) argument. [...]
> adding currentPos makes it much easier to understand the current
> execution state when debugging with GDB.

Makes sense -- nfa_match() has only the state list in scope, no row
index, which is exactly the thing that is hard to place in GDB. I will
thread currentPos through.

> The comments above in nfa_match are unnecessary because in ``if
> (RPRElemIsAbsorbableBranch(elem)`` we already have many comments, and
> the meaning seems the same.

Agreed -- the loop-top overview's second sentence restates the inline
END-chain block. I will drop it and keep "Evaluate VAR elements against
current row" as a one-line lead.

> Similar to REG_DEBUG, we really need an RPR_DEBUG option. [...] verify
> that ctx->states is not a circular linked list [...] without affecting
> release builds.

Fair point, and there is some history: RPR_DEBUG did exist earlier as
debug logging, and I am the one who proposed removing it. My reason was
narrow, though. I lean on the debugger, and with AI assistance the cost of
inserting, watching, and then removing ad-hoc logs has dropped sharply --
so I treated those logs as ephemeral and optimized for a clean committed
tree rather than for someone reading execRPR.c cold.

For a new contributor that trade-off is the wrong one, and you have
convinced me: a retained set of observation points, plus the Assert-style
sanity checks you describe (non-circular state list, count and
context-ordering invariants), would genuinely lower the barrier to this
code. So I agree with the direction. Rather than settle on a shape now, I
would leave it open: which observation points and checks are actually
worth retaining behind #ifdef RPR_DEBUG can be picked out later as a
separate follow-up, and it is really Tatsuo's call since he suggested the
first version.

> <para> The SQL standard defines more subclauses: MEASURES and SUBSET
> [...] more variations in AFTER MATCH clause. </para>
> This part should stay in the Compatibility section.

Agreed. It is a standard-deviation note, and the SELECT Compatibility
section currently has no row-pattern text, so I will move this paragraph
there.

Unless Tatsuo has a different view by the weekend, I will proceed with the
concrete changes above, leaving the RPR_DEBUG shape open as noted.

Best regards,
Henson

  427. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-07-02T00:23:30Z

    Hi Tatsuo,

Thanks for the review. I agree with the direction -- the (void) is
confusing. I did try your exact suggestion and the regression tests pass
unchanged, but I would keep one guard, for the following reason.

> Is there any reason that we call row_is_in_reduced_frame() here,
> instead of something like:
>
>    update_frameheadpos(winstate);
>    update_reduced_frame(winobj, pos);

row_is_in_reduced_frame() guards the update:

    state = get_reduced_frame_status(winstate, pos);
    if (state == RF_NOT_DETERMINED)
    {
        update_frameheadpos(winstate);
        update_reduced_frame(winobj, pos);
    }

Calling update_reduced_frame() unconditionally drops that guard. When
currentpos is already determined -- e.g. an interior row of a previous
match under AFTER MATCH SKIP PAST LAST ROW (RF_SKIPPED) -- it takes an
O(1) early return and overwrites the shared rpr_match_* record, silently
reclassifying a SKIPPED row as UNMATCHED.

This is not observable today -- I confirmed with a guarded-vs-unguarded
differential (patterns, both skip modes, various aggregates: identical
output), because both paths end up with an empty reduced frame. But it
does change the state semantics and leans on that convergence, so I would
rather keep the guard. Two ways, both of which remove the (void):

  A) Keep the guard inline at the call site.

  B) Factor it into a small helper shared by the nav-driving call site
     and row_is_in_reduced_frame():

        static void
        ensure_reduced_frame(WindowObject winobj, int64 pos)
        {
            WindowAggState *winstate = winobj->winstate;

            if (get_reduced_frame_status(winstate, pos) ==
RF_NOT_DETERMINED)
            {
                update_frameheadpos(winstate);
                update_reduced_frame(winobj, pos);
            }
        }

I lean towards B (no duplicated guard, and the name states the intent),
but A is the smaller diff. Which do you prefer?

Best regards,
Henson

  428. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-07-02T00:48:13Z

    Hi Henson,

> Hi Tatsuo,
> 
> Thanks for the review. I agree with the direction -- the (void) is
> confusing. I did try your exact suggestion and the regression tests pass
> unchanged, but I would keep one guard, for the following reason.
> 
>> Is there any reason that we call row_is_in_reduced_frame() here,
>> instead of something like:
>>
>>    update_frameheadpos(winstate);
>>    update_reduced_frame(winobj, pos);
> 
> row_is_in_reduced_frame() guards the update:
> 
>     state = get_reduced_frame_status(winstate, pos);
>     if (state == RF_NOT_DETERMINED)
>     {
>         update_frameheadpos(winstate);
>         update_reduced_frame(winobj, pos);
>     }
> 
> Calling update_reduced_frame() unconditionally drops that guard. When
> currentpos is already determined -- e.g. an interior row of a previous
> match under AFTER MATCH SKIP PAST LAST ROW (RF_SKIPPED) -- it takes an
> O(1) early return and overwrites the shared rpr_match_* record, silently
> reclassifying a SKIPPED row as UNMATCHED.
> 
> This is not observable today -- I confirmed with a guarded-vs-unguarded
> differential (patterns, both skip modes, various aggregates: identical
> output), because both paths end up with an empty reduced frame. But it
> does change the state semantics and leans on that convergence, so I would
> rather keep the guard. Two ways, both of which remove the (void):

Thanks for the explanation. That makese sense.

>   A) Keep the guard inline at the call site.
> 
>   B) Factor it into a small helper shared by the nav-driving call site
>      and row_is_in_reduced_frame():
> 
>         static void
>         ensure_reduced_frame(WindowObject winobj, int64 pos)
>         {
>             WindowAggState *winstate = winobj->winstate;
> 
>             if (get_reduced_frame_status(winstate, pos) ==
> RF_NOT_DETERMINED)
>             {
>                 update_frameheadpos(winstate);
>                 update_reduced_frame(winobj, pos);
>             }
>         }
> 
> I lean towards B (no duplicated guard, and the name states the intent),
> but A is the smaller diff. Which do you prefer?

I like B too.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  429. Re: Row pattern recognition

    jian he <jian.universality@gmail.com> — 2026-07-02T04:14:55Z

    Hi,

I have 3 patches to present:

v50-0001-refactor-DEFINE-volatile-expression-check.nocfbot
v50-0002-refactor-define_walker.nocfbot
v50-0003-replace-EMPTY_LOOP-to-RPR_ELEM_EMPTY_LOOP.nocfbot

As mentioned before, it's based on
https://github.com/assam258-5892/postgres/commits/RPR.
v50-0001, v50-0002 alrady submitted once patch at [1], now it's more complete.

Except that i want to rename RPSkipTo to RPRSkipTo and rpSkipTo to rprSkipTo,
What do you think?

[1] https://postgr.es/m/CACJufxG57=ddtbN=5RZCzhxWDYXvocKmB7NtZy+DoqZuhxb_DA@mail.gmail.com



--
jian
https://www.enterprisedb.com/

  430. Re: Row pattern recognition

    jian he <jian.universality@gmail.com> — 2026-07-03T00:36:49Z

    Hi.

``The actual mark is set to: min(lookback_mark, lookahead_mark).``
in README.rpr need update?

The attached patch is still based on
https://github.com/assam258-5892/postgres/commits/RPR.

v50-0001-Refactor-visit_nav_plan.partial
v50-0002-Refactor-visit_nav_exec.partial
v50-0003-no-need-compile-and-evaluate-offset-in-expression-evaluation.partial
v50-0004-misc.partial
v50-0001-Evaluate-navigation-offset-once-at-one-place.nocfbot

The ".partial" patch, when combined, contains the same content as the
single patch (v50-0001-Evaluate-navigation-offset-once-at-one-place.nocfbot).
(I first created separated patches, but then I felt the whole change
focuses on one main topic, so merging them made sense to me).

The commit message for the main patch
(v50-0001-Evaluate-navigation-offset-once-at-one-place.nocfbot) is
below:
------------------------------------------------
Subject: [PATCH v51 1/1] Evaluate navigation offset once at one place

Row pattern navigation (PREV/NEXT/FIRST/LAST and their compounds) resolved
offsets (transform a expression to a int64 number) in ExecInitExprRec,
ExecEvalRPRNavSet, extract_const_offset, eval_define_offsets. Serval places
transform the same offset expression to a int64 number, that is too much!

With the attached, a single place (ExecInitWindowAgg->eval_define_offsets) to
handling this.

visit_nav_exec() now evaluates every offset expression at executor init and save
the result on the RPRNavExpr (offset / compound_offset); ExecEvalRPRNavSet()
simply reads those int64 values.  This removes:

  - the RPRNavOffsetKind enum and the offset/kind fields on the
    WindowAgg plan node.  The planner walk (visit_nav_plan /
    compute_define_metadata) now only classifies match_start
    dependency, which is all buildRPRPattern() needs to decide context
    absorption.

  - the per-invocation offset machinery in the executor: the Datum*
    offset arrays in the eval step, rpr_nav_get_compound_offset(), and
    the scattered NULL/negative checks, now centralized in
    eval_nav_offset.

The "retain all" case (backward reach overflows int64) is encoded as the
sentinel navMaxOffset < 0 rather than an enum value; advance_nav_mark() disables
tuplestore trim on it.  EXPLAIN reads the resolved offsets from the planstate
(ExecInitWindowAgg is reached for EXPLAIN without ANALYZE), so it now prints
concrete offsets instead of "runtime".

```
 We must still set nav_saved_outertuple (done above) so that
EEOP_RPR_NAV_RESTORE is a harmless no-op.
```
The above in ExecEvalRPRNavSet is wrong, EEOP_RPR_NAV_RESTORE will never be a
no-op. However we can make EEOP_RPR_NAV_RESTORE no-op, so the above comments
make sense.

Rename eval_nav_offset_helper to eval_nav_offset

Based on https://github.com/assam258-5892/postgres/commits/RPR
------------------------------------------------



--
jian
https://www.enterprisedb.com/

  431. Re: Row pattern recognition

    jian he <jian.universality@gmail.com> — 2026-07-03T03:55:43Z

    On Fri, Jul 3, 2026 at 8:36 AM jian he <jian.universality@gmail.com> wrote:
>
> Hi.
>
> ``The actual mark is set to: min(lookback_mark, lookahead_mark).``
> in README.rpr need update?
>
> The attached patch is still based on
> https://github.com/assam258-5892/postgres/commits/RPR.
>
> v50-0001-Refactor-visit_nav_plan.partial
> v50-0002-Refactor-visit_nav_exec.partial
> v50-0003-no-need-compile-and-evaluate-offset-in-expression-evaluation.partial
> v50-0004-misc.partial
> v50-0001-Evaluate-navigation-offset-once-at-one-place.nocfbot
>
> The ".partial" patch, when combined, contains the same content as the
> single patch (v50-0001-Evaluate-navigation-offset-once-at-one-place.nocfbot).
> (I first created separated patches, but then I felt the whole change
> focuses on one main topic, so merging them made sense to me).
>
> The commit message for the main patch
> (v50-0001-Evaluate-navigation-offset-once-at-one-place.nocfbot) is
> below:
> ------------------------------------------------
Hi.

While at it, I found nav_traversal_walker is a little bit of
complicated (overkill).
So the attached patch is removing nav_traversal_walker.

Two files are attached:
v50-0001-Evaluate-navigation-offset-once-at-one-place.nocfbot
v50-0002-Drop-the-shared-nav_traversal_walker-for-RPR-DEFINE.nocfbot

v50-0001 is the same as posted in [1].
v50-0002 is built on top of v50-0001. Even without v50-0001,
I think the changes in v50-0002 still make sense, as they improve readability.

The commit message for v50-0002 is below:
------------------------------------------------
Subject: [PATCH v50 2/2] Drop the shared nav_traversal_walker for RPR DEFINE

Both visit_nav_exec() and visit_nav_plan() assert:

    Assert(nav->arg == NULL || !IsA(nav->arg, RPRNavExpr));
    Assert(nav->offset_arg == NULL || !IsA(nav->offset_arg, RPRNavExpr));
    Assert(nav->compound_offset_arg == NULL ||
           !IsA(nav->compound_offset_arg, RPRNavExpr));

so an RPRNavExpr is never nested inside another RPRNavExpr, and the expressions
that contain RPRNavExpr nodes are not complicated.  For that, the generic
nav_traversal_walker() -- a NavTraversal struct carrying a function pointer and
a void* context, shared between the planner and the executor is kind of
complicated.

Replace it with static RPRNavExpr_walker() in each site.  RPRNavExpr_walker will
finds RPRNavExpr nodes, and acts on them the way it wants.  Having a small
duplicated static walker in the two files is fine here, nodeWindowAgg.c no
longer need to include "optimizer/rpr.h".

rename visit_nav_plan to compute_matchStartDependent,
rename visit_nav_exec to compute_nav_offsets.
------------------------------------------------
[1]: https://www.postgresql.org/message-id/CACJufxFfMJAMc0R6g4f7Bwr47QDY%2BOBr2vzqqMRbCzjdD9vR2g%40mail.gmail.com



--
jian
https://www.enterprisedb.com/

  432. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-07-03T12:13:09Z

    Hi Henson, Jian,

In v50-0006-tidy-plumbing.patch, the planner cost model seems slightly
changed.  Before the cost was charged according to the number of
pattern variables in PATTERN clause.  But now it is charged according
to the number of pattern variables in DEFINE clause.  Maybe I missed
the discussion on the changing. Can you please explain the reason of
the change?

--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -3231,30 +3231,18 @@ cost_windowagg(Path *path, PlannerInfo *root,
 	 * so we'll just do this for now.
 	 *
 	 * Moreover, if row pattern recognition is used, we charge the DEFINE
-	 * expressions once per tuple for each variable that appears in PATTERN.
+	 * expressions once per tuple for each DEFINE variable.
 	 */
 	if (winclause->rpPattern)
 	{
-		List	   *pattern_vars;
 		QualCost	defcosts;
 
-		pattern_vars = collectPatternVariables(winclause->rpPattern);
-
-		foreach_node(String, pv, pattern_vars)
+		foreach_node(TargetEntry, def, winclause->defineClause)
 		{
-			char	   *ptname = strVal(pv);
-
-			foreach_node(TargetEntry, def, winclause->defineClause)
-			{
-				if (!strcmp(ptname, def->resname))
-				{
-					cost_qual_eval_node(&defcosts, (Node *) def->expr, root);
-					startup_cost += defcosts.startup;
-					total_cost += defcosts.per_tuple * input_tuples;
-				}
-			}
+			cost_qual_eval_node(&defcosts, (Node *) def->expr, root);
+			startup_cost += defcosts.startup;
+			total_cost += defcosts.per_tuple * input_tuples;
 		}

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  433. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-07-03T22:22:10Z

    Hi Tatsuo,

> In v50-0006-tidy-plumbing.patch, the planner cost model seems slightly
> changed. Before the cost was charged according to the number of
> pattern variables in PATTERN clause. But now it is charged according
> to the number of pattern variables in DEFINE clause. Maybe I missed
> the discussion on the changing. Can you please explain the reason of
> the change?

The change is a simplification that came out of Jian's v48 review, and it
does not change the cost the model produces.

It follows Jian's suggestion from 2026-06-15 [1]:

> collectPatternVariables is not needed.
> The parser already ensures every DEFINE variable appears in PATTERN,
> so there is nothing to filter.
> Also, we don't really do anything special (like make a dummy Const)
> regarding PATTERN variables that not appearing in the DEFINE clause.

The two loops charge exactly the same set, for the following reason:

- The old loop walked the unique PATTERN variables (collectPatternVariables
  deduplicates, returning each name once) and, for each, looked up the
  matching DEFINE entry by resname and charged that DEFINE's cost.
- Every DEFINE variable is guaranteed to appear in PATTERN -- the parser
  rejects a DEFINE variable that is not used in PATTERN (errmsg "DEFINE
  variable \"%s\" is not used in PATTERN" in parse_rpr.c).  So the DEFINE
  clause is always a subset of the unique PATTERN variables, and each
  DEFINE resname is unique.
- A PATTERN variable that has no DEFINE contributes nothing to the old
  loop, because the inner resname lookup finds no match.

So the old loop already charged each DEFINE expression exactly once, and
nothing else.  Iterating defineClause directly, as v50-0006 does, visits
precisely that same set once each.  The estimate is unchanged; only the
redundant outer walk over PATTERN and the per-variable resname lookup into
the DEFINE clause are removed.

This also matches the premise of the cost model we settled on back in
February: the NFA executor evaluates every DEFINE expression once per row,
so the natural unit for the per-tuple charge is the DEFINE variable.

[1]
https://www.postgresql.org/message-id/CACJufxFAQhbOD9EVCTAy-VwDbG4446N10GsxCcgdpFnjHO1Efw%40mail.gmail.com

Best regards,
Henson

  434. Re: Row pattern recognition

    jian he <jian.universality@gmail.com> — 2026-07-04T01:22:24Z

    On Sat, Jul 4, 2026 at 6:22 AM Henson Choi <assam258@gmail.com> wrote:
>
> Hi Tatsuo,
>
> > In v50-0006-tidy-plumbing.patch, the planner cost model seems slightly
> > changed. Before the cost was charged according to the number of
> > pattern variables in PATTERN clause. But now it is charged according
> > to the number of pattern variables in DEFINE clause. Maybe I missed
> > the discussion on the changing. Can you please explain the reason of
> > the change?
>
> The change is a simplification that came out of Jian's v48 review, and it
> does not change the cost the model produces.
>
> It follows Jian's suggestion from 2026-06-15 [1]:
>
> ....
>
> This also matches the premise of the cost model we settled on back in
> February: the NFA executor evaluates every DEFINE expression once per row,
> so the natural unit for the per-tuple charge is the DEFINE variable.
>
> [1] https://www.postgresql.org/message-id/CACJufxFAQhbOD9EVCTAy-VwDbG4446N10GsxCcgdpFnjHO1Efw%40mail.gmail.com
>
> Best regards,
> Henson

I am wondering if we can have one query to enable (COSTS ON) for testing.

For many rpr tests, the table size is small, EXPLAIN ANALYZE won't
take long, and probably all rows will be sampled.
Of course, once the patch is in committable shape, we need to switch it to OFF.

  435. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-07-04T07:28:41Z

    > The change is a simplification that came out of Jian's v48 review, and it
> does not change the cost the model produces.
> 
> It follows Jian's suggestion from 2026-06-15 [1]:
> 
>> collectPatternVariables is not needed.
>> The parser already ensures every DEFINE variable appears in PATTERN,
>> so there is nothing to filter.
>> Also, we don't really do anything special (like make a dummy Const)
>> regarding PATTERN variables that not appearing in the DEFINE clause.
> 
> The two loops charge exactly the same set, for the following reason:
> 
> - The old loop walked the unique PATTERN variables (collectPatternVariables
>   deduplicates, returning each name once) and, for each, looked up the
>   matching DEFINE entry by resname and charged that DEFINE's cost.
> - Every DEFINE variable is guaranteed to appear in PATTERN -- the parser
>   rejects a DEFINE variable that is not used in PATTERN (errmsg "DEFINE
>   variable \"%s\" is not used in PATTERN" in parse_rpr.c).  So the DEFINE
>   clause is always a subset of the unique PATTERN variables, and each
>   DEFINE resname is unique.
> - A PATTERN variable that has no DEFINE contributes nothing to the old
>   loop, because the inner resname lookup finds no match.
> 
> So the old loop already charged each DEFINE expression exactly once, and
> nothing else.  Iterating defineClause directly, as v50-0006 does, visits
> precisely that same set once each.  The estimate is unchanged; only the
> redundant outer walk over PATTERN and the per-variable resname lookup into
> the DEFINE clause are removed.

Ok, thanks for the explanation.

> This also matches the premise of the cost model we settled on back in
> February: the NFA executor evaluates every DEFINE expression once per row,
> so the natural unit for the per-tuple charge is the DEFINE variable.

BTW, I was thinking about cases where same DEFINE variable appears
twice or more in PATTERN for a same row. For example PATTERN
(A|A). But in this case it would be optimized out to (A). So we don't
need to worry about A appearing twice. So our cost model is correct in
this case.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  436. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-07-04T08:59:34Z

    Hi Tatsuo,

> BTW, I was thinking about cases where same DEFINE variable appears
> twice or more in PATTERN for a same row. For example PATTERN (A|A).
> But in this case it would be optimized out to (A). So we don't need
> to worry about A appearing twice. So our cost model is correct in
> this case.

Right, and I think the cost model holds even a bit more broadly than
the (A|A) case.  (A|A) does collapse to (A), but a variable can still
appear more than once in patterns that do not collapse, e.g. PATTERN
(A B | A C), or A A, or A+.

In those cases the model is still correct, for a slightly different
reason than the (A|A) rewrite: the executor evaluates each DEFINE
predicate once per row, not once per PATTERN occurrence.  For each
row it evaluates every DEFINE once and keeps the boolean results in
a varMatched array.

When the same A appears at several positions in the pattern -- for
example the A in each branch of (A B | A C), which are distinct
states -- each looks up varMatched[A], so the same entry is read
more than once; but that read reuses the already-computed value, not
a re-evaluation.  So repetition in PATTERN never multiplies DEFINE
evaluations, and charging once per DEFINE variable in the cost model
matches what the executor actually does.

The related question that does run the other way is that today we
evaluate every DEFINE for a row eagerly, not just the ones that row
actually needs.  For example, in PATTERN (A B C D) a single match
walks the sequence one variable per row -- each row only needs to
test the single variable its state expects -- yet we still evaluate
A, B, C, and D at every row.

That is the short-circuit / lazy DEFINE evaluation Jian raised on
2026-05-26 using that very (A B C D) example (evaluate a predicate
only the first time a state tests it).  If we ever adopt it, the
cost model's premise -- every DEFINE once per row -- would change
with it, so the two are tied together.

There's also a soundness angle that argues for keeping it separate.
DEFINE already forbids volatile functions and sequence operations
(nextval), so the obvious non-deterministic cases are out.  The
wrinkle lazy evaluation adds is that a predicate would then be
evaluated zero or one times per row -- skipped whenever no state
reaches it -- rather than always.  Whether that is safe for a
predicate carrying some state-affecting behavior the volatility ban
does not exclude is something I haven't worked through, so it wasn't a
call I'd want to make lightly under the current review.

As we discussed, that one is best left as a separate series after
the initial commit, and since it was Jian's idea I'd be glad to see
him drive it.  For now I'd keep it out of the in-flight review so the
commit stays small.

Best regards,
Henson

  437. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-07-04T16:39:43Z

    Hi hackers,

This is a design note on the core data structures for CLASSIFIER and
row pattern variables, aimed at the next round of window-clause row
pattern recognition (RPR, R020) rather than the current patch. I'd
like to share it and open it up for discussion.

It opens with a reflection, because the reflection is the point. For
about six months I took it for granted that we must keep the full
per-row match history during matching. That premise was wrong. Once I
dropped it, the data-structure problem dissolved into a handful of
small, bounded pieces -- which is what the rest of the note works out:
what to store per case, how much, and how to encode it.

I believe the storage/representation side is settled. What I have
deliberately left open is the part that lives in the matching engine
-- the aggregate accumulator lifecycle (clone on branch, discard on
failure) and the winning-match replay. I'd welcome thoughts there,
particularly from Tatsuo and Jian, given their work on the current
patch.

Best regards,
Henson

----------------------------------------------------------------------

RPR CLASSIFIER / row pattern variable storage (window clause)

I. Reflection -- the "full match history" premise was wrong

Since January this year -- six months into starting RPR work in
earnest -- I kept reasoning under one premise:

    To implement CLASSIFIER and row pattern variables, we must keep
    the per-row sequence of variable labels (the match history) that
    the match consumes, throughout matching.

For that half year I set the data-structure problem on top of it --
"where and how do we hold that whole sequence cheaply?" -- and kept
piling up storage / sharing / compression / lookup structures.

Yet no usable lookup structure ever came out. Worse: matchStates
branch and most of them vanish on match failure, so the cost of
building a lookup structure for each of those countless
(soon-discarded) states was exceeding the lookup efficiency it bought
-- paying up front, for the many that die, on behalf of the few that
survive.

The premise was wrong. RPR does not need to keep the full match
history during matching. The one place a full per-row sequence is
genuinely materialized is the universal-scope MEASURE aggregate (case
8, ARRAY_AGG(CLASSIFIER())). So that entire storage / sharing /
compression / lookup apparatus was solving a problem that did not need
solving.

II. The corrected picture -- what to store, per case

Here is how the storage requirement splits, case by case:

1. DEFINE navigation -- the reference range is bounded by the
   navigation offset. The offset is a run-time constant, so the upper
   bound to keep during execution is fixed and can be computed
   directly, without any lookup. A nav with no row pattern variable
   just derives the position from the offset and reads it from the
   tuplestore. (How much to retain, per nav function, is detailed in
   section III.)

2. bare row pattern variable reference (A.col) -- equivalent to
   LAST(A.col, 0). The structure is a pos list for A -- since
   everything in it is A, membership is implicit and no varId is
   needed. As a bare reference this is the special case with length =
   1 (only the last pos); the full list is needed only for an A
   aggregate or a larger FIRST/LAST offset (section III).

3. SUBSET (union) variable -- for SUBSET U=(A,B), CLASSIFIER(U) and
   U.col are merely derived from the primary classification (which row
   is A/B); they are not new information. Because A and B are
   interleaved in the list and we must tell them apart, the structure
   is a list of (pos, varId) pairs.

4. bare CLASSIFIER() (current row) -- unless it is inside an aggregate
   (ARRAY_AGG etc.), only the current row is needed (no list). The
   classifier comes straight from the variable (its name) that the
   pattern element currently being consumed by the matchState
   designates.

5. aggregate in DEFINE -- it drives matching, so it is evaluated live
   during matching under RUNNING semantics (no post-processing
   possible). The aggregate inputs (which row is which variable, its
   column, its classifier) are all already obtained from 1-4, so no
   new structure is needed -- only one running accumulator folding
   them on top. That is as far as the match-history view goes (no full
   history needed). Separately, there are two costs: (a) computing
   live for the many matchStates that mostly fail, and (b) cloning the
   in-progress accumulator into children on a branch. But this is
   aggregate accumulator management cost, not match history.

6-8 (MEASURE aggregates): since the match is already complete, it is
better to avoid paying for a live accumulator per failing candidate
(the cost in 5) and instead process only the winning match by post
replay. What each case must store/replay is below.

6. aggregate in MEASURE -- variable/SUBSET scope -- e.g. sum(A.price),
   sum(U.price), ARRAY_AGG(CLASSIFIER(U)). Only membership is needed
   (values in the tuplestore), compressed as a bitmap / run length
   (alternating m A's and n non-A's; sequential, no lookup). But the
   granularity differs: sum(A.price) / sum(U.price) need only is-A /
   is-U, whereas ARRAY_AGG(CLASSIFIER(U)) must distinguish which
   sub-variable, so it needs per-sub-variable membership (A and B
   each). Those per-sub memberships are precisely the per-row member
   label, and when several aggregates use the same sub-variable, these
   memberships fuse into that single label. (For bare scalar
   CLASSIFIER(U), see 3 -- just the last varId in the U list.)

7. aggregate in MEASURE -- CLASSIFIER(variable) inside an aggregate
   (special case: optimized to the first-occurrence position) -- in
   ARRAY_AGG(CLASSIFIER(A)), CLASSIFIER(A) is a running LAST, so its
   value is null before the first A and the constant 'A' afterwards.
   So instead of the full membership of 6, it optimizes to a single
   first-occurrence position for A (the null->A boundary) -- a
   single-variable special case. (For SUBSET CLASSIFIER(U) the member
   varies, so this optimization does not apply; handle it in 6.) bare
   scalar CLASSIFIER(A) is derived from 2 -- just whether A's list is
   empty.

8. aggregate in MEASURE -- universal scope -- every match row is in
   scope. For a column (e.g. sum(price)) it is all rows, so not even
   membership is needed (values in the tuplestore). For
   ARRAY_AGG(CLASSIFIER()), each row's value is the row pattern
   variable name (the label), and the storage is a run-length encoding
   of varId (a run of the same variable as (varId, run)) -- since all
   rows are in order, no pos is needed; just pile up varId runs (order
   implies pos). (For bare scalar CLASSIFIER(), see 4 -- just the last
   row's varId.)

III. Retained length per nav function -- offset fixes the bound

If 1-2 above say what to hold, this pins down how much of it to hold
during matching, per nav function. The key: the offset is a run-time
constant (a literal, embedded variable, and the like -- not a column
or subquery), so the size of the retained window is fixed at plan
time. So you can pre-allocate a ring buffer of that size per variable,
and there is no lookup during matching.

Physical nav and logical nav split apart:

- physical nav (PREV/NEXT) -- the offset is a physical position in the
  partition, so it does not grow the variable's history at all. The
  anchor is just "that variable's running last row", and offset n
  merely indexes the tuplestore from there. Even when NEXT looks
  ahead, that row is already in the tuplestore (only its
  classification is pending). So: retain = 1 anchor, extra +0.

- logical nav (FIRST/LAST) -- the offset is a position within the rows
  mapped to that variable. Their retention costs are asymmetric.
  FIRST(X.col, n) is the (n+1)-th from the front, and the front never
  changes once passed, so you count X rows and latch only the single
  row where the count reaches n+1 -- a counter + 1 holder, O(1)
  regardless of n (later X rows are ignored). LAST(X.col, n) is the
  (n+1)-th from the back, but since it is running the end keeps
  moving, so you must hold X's last n+1 rows in a sliding window --
  retain = n+1. bare X.col = LAST(X.col, 0) = 1.

- nested PREV/NEXT(FIRST/LAST(X.col, k), m) -- retention follows the
  rule above for the inner (logical) part only: counter+holder for
  FIRST, k+1 sliding for LAST. The physical part m just reads the
  tuplestore from that row, so +0.

Boundary -- this bound applies to navigation references only. An
aggregate reference (sum(A.price) etc.) folds the entire prefix, so it
is absorbed into a running accumulator (5), not a window -- an O(1)
accumulated value, not a length. Do not mix the two into one
structure.

IV. Compressed storage -- one run-length spine

2-3 (during matching) and 6-8 (MEASURE replay) use the same
information under different access patterns, so their encodings
diverge:

- live / sparse (2-3) -- during matching you append only the rows
  mapped to X, as they come. It is a subsequence of all rows, so to
  know which row is which you must carry the pos too (2's pos list,
  3's pos+varId).

- replay / dense (6-8) -- you walk the winning match from the start
  over every match row, so no pos is needed -- order is the pos. This
  is where compression pays off.

The replay-side structures are all one run-length spine, specialized
only by granularity:

- universal classifier (8) -- per-row varId as (varId, run) runs. No
  pos; the cumulative run lengths give the position.

- per-variable membership (6) -- is-X as alternating runs: just a
  repetition of (present length, absent length) pairs (alternative: a
  1-bit-per-row bitmap). The case where is-A alone suffices, as in
  sum(A.price).

- per-sub-member (6) -- (varId, run) within the union, when you must
  tell A/B apart as in ARRAY_AGG(CLASSIFIER(U)). Rows outside U
  (non-A/non-B) must be carried as runs of a "none" sentinel varId so
  that order = pos is preserved.

- single-variable boundary (7) -- ARRAY_AGG(CLASSIFIER(A)) has only
  the null->A boundary, so the run degenerates to essentially a single
  item.

single varId RLE vs several per-variable memberships -- for the
per-sub-member case the two encodings trade off. (i) single varId RLE
(the bullet above): the per-row label comes out directly, but every
aggregate scans the whole span. (ii) several per-variable memberships:
is-A and is-B each as alternating runs (no sentinel needed) -- an
aggregate can pick just the sub-variable streams it uses, but a
CLASSIFIER(U) label must be read by merging several streams.

bitmap vs RLE -- a match is inherently runs (the greedy A^m B^n
shape). So RLE is natural, and since consumption is a sequential scan
no index is needed. A 1-bit-per-row bitmap (as long as the match) wins
only when the runs fragment finely or random access is needed.

Values are not stored -- the spine carries only
membership/label(varId); the actual column values live in the
tuplestore, and the replay scan fetches them by pos as it goes. This
is the principle behind 6-8 repeatedly noting that values stay in the
tuplestore.

  438. Re: Row pattern recognition

    Tatsuo Ishii <ishii@postgresql.org> — 2026-07-06T06:02:21Z

    Hi Henson,

> In those cases the model is still correct, for a slightly different
> reason than the (A|A) rewrite: the executor evaluates each DEFINE
> predicate once per row, not once per PATTERN occurrence.  For each
> row it evaluates every DEFINE once and keeps the boolean results in
> a varMatched array.
> 
> When the same A appears at several positions in the pattern -- for
> example the A in each branch of (A B | A C), which are distinct
> states -- each looks up varMatched[A], so the same entry is read
> more than once; but that read reuses the already-computed value, not
> a re-evaluation.  So repetition in PATTERN never multiplies DEFINE
> evaluations, and charging once per DEFINE variable in the cost model
> matches what the executor actually does.
> 
> The related question that does run the other way is that today we
> evaluate every DEFINE for a row eagerly, not just the ones that row
> actually needs.  For example, in PATTERN (A B C D) a single match
> walks the sequence one variable per row -- each row only needs to
> test the single variable its state expects -- yet we still evaluate
> A, B, C, and D at every row.
> 
> That is the short-circuit / lazy DEFINE evaluation Jian raised on
> 2026-05-26 using that very (A B C D) example (evaluate a predicate
> only the first time a state tests it).  If we ever adopt it, the
> cost model's premise -- every DEFINE once per row -- would change
> with it, so the two are tied together.
> 
> There's also a soundness angle that argues for keeping it separate.
> DEFINE already forbids volatile functions and sequence operations
> (nextval), so the obvious non-deterministic cases are out.  The
> wrinkle lazy evaluation adds is that a predicate would then be
> evaluated zero or one times per row -- skipped whenever no state
> reaches it -- rather than always.  Whether that is safe for a
> predicate carrying some state-affecting behavior the volatility ban
> does not exclude is something I haven't worked through, so it wasn't a
> call I'd want to make lightly under the current review.
> 
> As we discussed, that one is best left as a separate series after
> the initial commit, and since it was Jian's idea I'd be glad to see
> him drive it.  For now I'd keep it out of the in-flight review so the
> commit stays small.

Agreed.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

  439. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-07-07T03:40:24Z

    Hi Jian,

Thanks for this -- consolidating the offset resolution into one place
is a direction I like, and a couple of things about it are genuinely
nice:

> EXPLAIN reads the resolved offsets from the planstate
> (ExecInitWindowAgg is reached for EXPLAIN without ANALYZE), so it now
> prints concrete offsets instead of "runtime".

That is a real improvement -- the offset becomes visible in the plan
even when it comes from a bind parameter, where today we can only show
"runtime".

It also lines up with a point Tatsuo made back in April: since the
planner already folds a constant offset expression down to a single
Const, doing the resolution in the executor means we no longer have to
care whether the offset was constant or not.  So the direction has his
backing as well.

That said, I don't think we should take it into the patch at this
stage.  The current version has one proven defect: a plain
FIRST(expr, n) with n > 0 loses its forward reach.  I added a test for
it -- "DEFINE A AS v > FIRST(v, 5)" in rpr_explain -- where the
baseline prints "Nav Mark Lookahead: 5" and the patch prints 0.  The
query result is still correct (the mark is under-advanced, so it only
keeps extra rows), but the EXPLAIN is wrong and the tuplestore trim is
lost, so it is a regression against the baseline.

The test will come with the next patch email I send, so you will have
it there to diff against.

More broadly, though, reshaping the offset handling this much right as
we are stabilizing for commit feels like the wrong time -- at this
point I would rather limit changes to minimal fixes for proven
defects.  This is a worthwhile cleanup that deserves proper review
time, and it would be more valuable if it can be refined without that
pressure.

So how about, once the current patch is verified and committed, we
pursue this as a separate follow-up?

Best regards,
Henson

  440. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-07-07T04:08:42Z

    Hi Jian,

My first reaction to the define_walker refactor was that something
about it did not sit right, but I could not put my finger on why, so I
kept putting off a clear answer -- sorry about that.  I have taken the
time to think it through now.

To be clear, the patch is good, idiomatic PostgreSQL code:
contain_var_clause() and contain_rprnavexpr() are exactly how we
express these checks elsewhere, and it trims a fair amount of code.

> The allowed nav-expression combination is limited, therefore,
> has_column_ref is not necessary.

This is where I land differently, though.  A DEFINE predicate is
context-sensitive: the same column reference is required inside a
navigation argument but rejected inside its offset.  In
"DEFINE A AS PREV(price, price) > 0", the first price (the argument)
is required and the second (the offset) is rejected.  The
phase-tracking walker models that context directly, in a single pass,
and it generalizes to whatever context-dependent nesting the clause
grows into (MEASURES, further navigation forms).  Dropping it works
today only because the allowed combinations are limited, as you note
-- it leans on that limit rather than on the shape of the clause.

It also gives up the single-pass property the walker was written for
("no subtree is walked twice"): each navigation argument and offset is
now walked about twice, once by the main walk and again by the
contain_* check.  The extra cost is small at parse time, but it is a
step back from a design that deliberately avoided it.

So my preference is to keep the context-aware walker as it stands.
This is more about which foundation we want for an inherently
context-sensitive clause than about anything being wrong in your
version -- if you see it the other way, I would be glad to hear it.

Best regards,
Henson

  441. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-07-07T07:07:29Z

    Hi Tatsuo, Jian,

I found a wrong result that comes from the context-absorption
optimization, and wanted to run it by you both.

  WITH d(id, a, c) AS (
    VALUES (1, true,  false),
           (2, true,  true),
           (3, true,  false),
           (4, false, false))
  SELECT id, count(*) OVER w AS cnt
  FROM d
  WINDOW w AS (
    ORDER BY id
    ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
    AFTER MATCH SKIP PAST LAST ROW
    PATTERN (A{5,} | C)
    DEFINE A AS a, C AS c);

   id | cnt
  ----+-----
    1 |   0
    2 |   0
    3 |   0
    4 |   0

id = 2 returns cnt = 0, but C matches [2,2] there, so it should be
1. EXPLAIN ANALYZE on the query reports "0 matched" and "2 absorbed".

The cause: whether a context can be absorbed is decided from its
in-progress states. Once the id=2 context records the C match [2,2],
that match moves to matchedState and its C state leaves the
in-progress set, so only the A run remains and the context is judged
fully absorbable. The id=1 context's longer A run then dominates it
and frees the whole context -- including the recorded [2,2] match.
The dominance argument only covers a context's future matches; it
does not account for a match already recorded on the non-absorbable
branch.

The fix: bring matchedState into the absorption comparison as well --
a context should be absorbed only when the absorbing context also
covers the recorded match, not just the in-progress states, so a
match the absorber cannot reproduce is never dropped.

Best regards,
Henson

  442. Re: Row pattern recognition

    Henson Choi <assam258@gmail.com> — 2026-07-08T08:04:34Z

    Hi Tatsuo, Jian,

I hit a wrong result in row pattern recognition: an alternation
whose last branch is a concatenation of quantified groups matches
one group short.

  WITH d(id, dd, ee) AS (VALUES
    (1, true,  false), (2, false, true),
    (3, true,  false), (4, false, true))
  SELECT id, count(*) OVER w AS cnt
  FROM d
  WINDOW w AS (
    ORDER BY id
    ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
    PATTERN (A | (B C)+ (D E)+)
    DEFINE A AS false, B AS false, C AS false,
           D AS dd, E AS ee);

   id | cnt
  ----+-----
    1 |   4      <- wrong; should be 0 (no match)
    2 |   0
    3 |   0
    4 |   0

The rows are D E D E, with no B and no C.  By precedence
PATTERN (A | (B C)+ (D E)+) means A | ((B C)+ (D E)+), so a bare
(D E)+ satisfies no branch and no row should match.  Instead row 1
matches [1,4]: the pattern behaves as if it were written
A | (B C)+ | (D E)+.

Cause: the ALT branch walk follows the branch head's jump to reach
the next branch.  That jump was the branch link, but the group
BEGIN's skip-past-END path was later layered onto the same field,
so jump is now overloaded.  When the last branch is (B C)+ (D E)+,
the BEGIN of (B C)+ jumps past its END to the BEGIN of (D E)+, and
the walk mistakes that following group for another alternative.

The same overloaded jump is walked in three places:
nfa_advance_alt (match advance), computeAbsorbabilityRecursive
(absorption marking), and the deparse.  nfa_advance_alt gives the
wrong result above.  computeAbsorbabilityRecursive over-marks the
trailing group as absorbable, though that turns out inert at run
time.  The deparse already sidesteps the overload in
rpr_next_branch() with the relative test elem[j-1].next != j, so
EXPLAIN prints the pattern correctly.

Fix direction: at its core, separate the BEGIN jump (group skip)
from the ALT branch-link jump -- they share one field today, which
is the overload.  I am weighing two ways to do that.

One is to move the group BEGIN's skip-past-END onto next.  Group
entry is already the contiguous BEGIN+1, so next is free, and jump
is left to be the branch link only.  No new elements.

The other is explicit branch-separator markers (a new SEP varid)
that carry the branch link, chained ALT.jump -> SEP -> ... ->
jump = -1 on the last one; branch content still reaches the
post-ALT element through next.

Either way the ALT walk sees jump as a branch link or -1, so
nfa_advance_alt, computeAbsorbabilityRecursive and the deparse can
enumerate branches the same clean way, dropping the relative test
and the depth-based break.  Deparse output stays the same.

I can put together a patch along these lines if the direction
looks right.

Best regards,
Henson