Horton

Re: BUG #18080: to_tsvector fails for long text input

Tom Lane <tgl@sss.pgh.pa.us>

View thread View on postgr.es
From: Tom Lane <tgl@sss.pgh.pa.us>
To: Alvaro Herrera <alvherre@alvh.no-ip.org>
Cc: uwe.binder@pass-consulting.com, pgsql-bugs@lists.postgresql.org
Date: 2023-09-15T13:53:49Z
Lists: pgsql-bugs 
Alvaro Herrera <alvherre@alvh.no-ip.org> writes:
> On 2023-Sep-04, PG Bug reporting form wrote:
>> SELECT to_tsvector('english'::regconfig, (REPEAT('<Long123456789/>'::text,
>> 20000000)));
>> results in
>> ERROR:  invalid memory alloc request size 2133333320

> This is because to_tsvector_byid does this:
>     prs.lenwords = VARSIZE_ANY_EXHDR(in) / 6;    /* just estimation of word's
>                                                  * number */
>     if (prs.lenwords < 2)
>         prs.lenwords = 2;

Yeah.  My thought about blocking the error had been to limit
prs.lenwords to MaxAllocSize/sizeof(ParsedWord) in this code.
I doubt that switching over to MCXT_ALLOC_HUGE is a good idea.
(Would we not also have to touch the places that repalloc that
array?)

			regards, tom lane

Commits

  1. Pack struct ParsedWord more tightly.

  2. Limit to_tsvector_byid's initial array allocation to something sane.