Thread

1. [PATCH] Make ReScanForeignScan callback optional for FDWs

   Adam Lee <adam8157@gmail.com> — 2025-12-03T08:45:32Z

   Hi hackers,
   
   I'd like to propose a patch that makes the ReScanForeignScan callback 
   optional for Foreign Data Wrappers, allowing the planner to automatically 
   handle non-rescannable FDWs.
   
   # Background
   
   Currently, FDWs are expected to implement ReScanForeignScan to support 
   rescanning in scenarios like nested loop joins. However, some FDWs just cannot
   implement rescanning:
   
   - Streaming data sources (Kafka, message queues)
   - One-time token-based APIs with expensive re-authentication
   - Data sources where re-fetching is prohibitively expensive
   - ...
   
   Right now, these FDWs either implement a stub ReScanForeignScan that fails 
   at runtime, or they buffer all data in BeginForeignScan to support rescan, 
   which wastes memory when rescanning isn't needed.
   
   # Proposed Solution
   
   This patch introduces a 'rescannable' field in the Path structure that 
   tracks whether each path supports rescanning. The planner uses this 
   information to:
   
   1. Auto-detect FDW rescan capability by checking if ReScanForeignScan is 
      provided
   2. Automatically insert Material nodes when non-rescannable paths are used 
      as inner paths in nested loops
   3. Reject parameterized foreign scan paths if the FDW doesn't support 
      rescan (preventing planning failures)
   4. Raise a clear error for correlated subqueries that cannot be handled 
      (these are planned independently and can't use Material nodes)
   
   # Not only FDWs
   
   Beyond enabling non-rescannable FDWs, this mechanism could also be used 
   for performance optimization on other nodes. Some operations can technically
   support rescan but at significant cost (MergeAppend redo all the sorts,
   Aggregation redo all the calculations...). We may could mark such paths as
   non-rescannable in some cases to encourage the planner to materialize results
   instead.
   
   The patch is attached. If reviewers feel this is a good idea and needs more
   discussion or the complexity warrants it, I'm happy to register this for the
   next CommitFest. For now, I'm sharing it here to gather initial feedback on the
   approach.
   
   -- 
   Adam
   

2. Re: [PATCH] Make ReScanForeignScan callback optional for FDWs

   Robert Haas <robertmhaas@gmail.com> — 2025-12-18T18:15:13Z

   On Wed, Dec 3, 2025 at 3:45 AM Adam Lee <adam8157@gmail.com> wrote:
   > Beyond enabling non-rescannable FDWs, this mechanism could also be used
   > for performance optimization on other nodes. Some operations can technically
   > support rescan but at significant cost (MergeAppend redo all the sorts,
   > Aggregation redo all the calculations...). We may could mark such paths as
   > non-rescannable in some cases to encourage the planner to materialize results
   > instead.
   
   I'm not 100% sure, but my first reaction is to think that adding a
   'rescannable' field to the Path struct is a bad idea. In these other
   cases that you mention, I think we'd want to consider the cost of
   paths with and without a materialize node and pick the cheaper one. We
   don't have a terribly robust model for the cost of rescanning things
   today, I think, but improving seems like the right way forward, rather
   than making a hard decision that a certain node fully support
   rescanning or is totally incapable of it.
   
   My suggestion would be to consider just adding another callback to
   FdwRoutine, or maybe giving one of the existing planner callbacks the
   additional job of deciding whether rescan is allowed. That's a much
   smaller design change. Suddenly making rescannability a property of
   every Path node of every kind seems like it could have broad-ranging
   ripple effects and I find that pretty scary. Just allowing an FDW to
   turn it off if it really can't implement that feels like it might be
   reasonable.
   
   -- 
   Robert Haas
   EDB: http://www.enterprisedb.com