Re: Incremental View Maintenance, take 2

On Fri, 12 Sep 2025 14:09:47 +0800
Zhang Mingli <zmlpostgres@gmail.com> wrote:

> Hi, all
> 
> On Aug 30, 2025 at 04:04 +0800, Yugo Nagata <nagata@sraoss.co.jp>, wrote:
> >
> > Anyway, I will start rebasing the patches, reorganizing the patch set,
> > and applying fixes made in pg_ivm [2].
> 
> First of all, we would like to express our sincere gratitude for your continuous efforts and contributions to the IVM feature.
> Our AQUMV (Answer Query Using Materialized Views) functionality in Apache Cloudberry is built directly upon your foundational work.
> 
> 
> We recently encountered an issue in one of our customer's production environments.
> The problem occurs in the function CreateIvmTriggersOnBaseTablesRecurse(), where a Bitmapset relidsis used to record rte->relid.
> 
> ```
> case T_RangeTblRef:
> {
>  int rti = ((RangeTblRef *) node)->rtindex;
>  RangeTblEntry *rte = rt_fetch(rti, qry->rtable);
> 
>  if (rte->rtekind == RTE_RELATION && !bms_is_member(rte->relid, *relids))
>  {
>  CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_INSERT, TRIGGER_TYPE_BEFORE, ex_lock);
>  CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_DELETE, TRIGGER_TYPE_BEFORE, ex_lock);
>  CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_UPDATE, TRIGGER_TYPE_BEFORE, ex_lock);
>  CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_TRUNCATE, TRIGGER_TYPE_BEFORE, true);
>  CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_INSERT, TRIGGER_TYPE_AFTER, ex_lock);
>  CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_DELETE, TRIGGER_TYPE_AFTER, ex_lock);
>  CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_UPDATE, TRIGGER_TYPE_AFTER, ex_lock);
>  CreateIvmTrigger(rte->relid, matviewOid, TRIGGER_TYPE_TRUNCATE, TRIGGER_TYPE_AFTER, true);
> 
>  *relids = bms_add_member(*relids, rte->relid);
>  }
> }
> ```
> However,  the Bitmapset structure is typically designed to store small integers (such as relation index), whereas rte->relidrepresents the Oid of a relation.
> For instance, when a table is created immediately after initializing a new cluster, its Oid might be 17019. Storing such a value in a Bitmapset consumes approximately 0.2MB of memory when creating an IVM.
> 
> (gdb) p bmsToString(((Bitmapset *) relids))
> $13 = 0x5643d7bbb070 "(b 17019)"
> (gdb) p *((Bitmapset *) relids)
> $14 = {nwords = 266, words = 0x5643d79d1bd8}
> 
> This memory usage becomes even more significant when dealing with larger Oids.
> Moreover, a more critical issue arises when the Oid exceeds 0x7FFFFFFF. Since Oid is an unsigned integer, but the Bitmapset expects signed integers, an assertion failure will occur in such cases.
> 
> 
> We have encountered similar requirements in our own scenarios and eventually switched to using a HTAB implementation. Hope this proves useful for your work.

Sorry for the late reply. While reviewing the discussion around reworking this, 
I noticed that I never replied to this comment.

That makes sense. I'll update the patch accordingly.

Regards,
Yugo Nagata

-- 
Yugo Nagata <nagata@sraoss.co.jp>