Re: Assert failure on 'list_member_ptr(rel->joininfo, restrictinfo)'

On 11/12/2023 09:31, Richard Guo wrote:
> On Fri, Dec 8, 2023 at 3:13 PM Alexander Pyhalov 
> <a.pyhalov@postgrespro.ru <mailto:a.pyhalov@postgrespro.ru>> wrote:
>     Andrei Lepikhov писал(а) 2023-12-08 07:37:
>      > I'd already clashed with Tom on copying the required_relids field
>     and
>      > voluntarily made unnecessary copies in the project [1].
>      > And ... stuck into huge memory consumption. The reason was in
>      > Bitmapsets:
>      > When we have 1E3-1E4 partitions and try to reparameterize a join,
>     one
>      > bitmapset field can have a size of about 1kB. Having bitmapset
>      > referencing Relation with a large index value, we had a lot of (for
>      > example, 1E4 * 1kB) copies on each reparametrization of such a
>     field.
>      > Alexander Pyhalov should remember that case.
>     Yes. If it matters, this happened during reparametrization when 2
>     partitioned tables with 1000 partitions each were joined. Then
>     asymmetric  pw join managed to eat lots of memory for bitmapsets (by
>     lots of memory I mean all available on the test VM).
> By reparametrization did you mean the work done in
> reparameterize_path_by_child()?  If so maybe you'd be interested in the
> patch [1] which postpones reparameterization of paths until createplan.c
> and thus can help avoid unnecessary reparametrization work.

Yeah, I have discovered it already. It is a promising solution and only 
needs a bit more review. But here, I embraced some corner cases with the 
idea that we may not see other cases right now. And also, sometimes the 
Bitmapset field is significant - it is not a corner case.

-- 
regards,
Andrei Lepikhov
Postgres Professional