Re: BUG #16925: ERROR: invalid DSA memory alloc request size 1073741824 CONTEXT: parallel worker

On Sat, Mar 13, 2021 at 11:21 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> On Sat, Mar 13, 2021 at 8:12 AM PG Bug reporting form
> <noreply@postgresql.org> wrote:
> > Using TPC-H benchmark scale-factor 1000.  To recreate dataset, see
> > https://github.com/swarm64/s64da-benchmark-toolkit.  Use psql_native
> > schema.
>
> I'd like to reproduce this but it may take me some time.  Can you
> please show the query plan?
>
> > 2021-03-12 19:45:37.352 CET [316243] ERROR:  XX000: invalid DSA memory alloc
> > request size 1073741824
>
> So, this means we have a call to dsa_allocate_extended() without the
> DSA_ALLOC_HUGE flag failing the sanity check that surely no one wants
> a GB of memory at once.  In hash joins, we deliberately avoid making
> our hash table bucket array long enough to hit that, since commit
> 86a2218e, and all other data is allocated in small chunks.  So the
> only way to hit this would be with an individual tuple that takes 1GB
> to store.  Other paths that use DSA include bitmap heap scans, but
> they use the DSA_ALLOC_HUGE flag at least in one place.
>
> >  max_parallel_workers                              | 1000
> >  max_parallel_workers_per_gather                   | 52

Another way to make a very large single allocation with a parallel
hash join with a large number of partitions and participants.  Are you
in a position to debug and test patched versions?  It'd be interesting
to know if ExecParallelHashJoinSetUpBatches() is the location, where
it does:

    pstate->batches =
        dsa_allocate0(hashtable->area,
                      EstimateParallelHashJoinBatch(hashtable) * nbatch);

If so, questions of whether it's really sane to run with so many
batches aside, the solution could be dsa_allocate_extended(...,
DSA_ALLOC_HUGE | DSA_ALLOC_ZERO).