Re: PG18 GIN parallel index build crash - invalid memory alloc request size

On Sun, Oct 26, 2025 at 5:52 PM Tomas Vondra <tomas@vondra.me> wrote:

> I like (a) more, because it's more consistent with how I understand m_w_m.
> It's weird
> to say "use up to 20GB of memory" and then the system overrides that with
> "1GB".
> I don't think it affects performance, though.
>

There wasn't really that much gain from 1GB -> 20GB, I was using that
setting for QA purposes more than measured performance.  During the early
parts of an OSM build, you need to have a big Node Cache to hit max speed,
1/2 or more of a ~90GB file.  Once that part finishes, the 45GB+ cache
block frees up and index building starts.  I just looked at how much was
just freed and thought "ehhh...split it in half and maybe 20GB maintenance
mem?"  Results seemed a little better than the 1GB setting I started at, so
I've ran with that 20GB setting since.

That was back in PG14 and so many bottlenecks have moved around.  Since
reporting this bug I've done a set of PG18 tests with m_w_m=256MB, and one
of them just broke my previous record time running PG17.  So even that size
setting seems fine.

I also wonder how far are we from hitting the uint32 limits. FAICS with
> m_w_m=24GB we might end up with too many elements, even with serial
> index builds. It'd have to be a quite weird data set, though.


Since I'm starting to doubt I ever really needed even 20GB, I wouldn't
stress about supporting that much being important.  I'll see if I can
trigger an overflow with a test case though, maybe it's worth protecting
against even if it's not a functional setting.

--
Greg Smith, Software Engineering
Snowflake - Where Data Does More
gregory.smith@snowflake.com