Horton

Re: BUG #15964: vacuumdb.c:187:10: error: use of undeclared identifier 'FD_SETSIZE'

Alvaro Herrera <alvherre@2ndquadrant.com>

View thread View on postgr.es
From: Alvaro Herrera <alvherre@2ndquadrant.com>
To: Andres Freund <andres@anarazel.de>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Dilip Kumar <dilipbalaut@gmail.com>, jungleboogie0@gmail.com, pgsql-bugs@lists.postgresql.org, Michael Paquier <michael@paquier.xyz>
Date: 2019-08-19T02:59:54Z
Lists: pgsql-bugs 
On 2019-Aug-17, Andres Freund wrote:

> Hm. This made me think: Why is
> 
>                 if (concurrentCons > FD_SETSIZE - 1)
>                 {
>                     pg_log_error("too many parallel jobs requested (maximum: %d)",
>                                  FD_SETSIZE - 1);
> 
> a useful test / error message?  FD_SETSIZE is about the numerical value
> of fds.  There will usually be at least three fds open, starting at 0 -
> but there easily can be more, depending on what the reindexdb/vacuumdb
> caller is doing.

Hmm ... yeah, this is clearly not perfect.  In my laptop, vacuumdb -j 1021
works; 1022 and 1023 fail like this after opening a number of conns:

vacuumdb: vacuuming database "alvherre"
vacuumdb: error: could not connect to database alvherre: could not look up local user ID 1000: Too many open files

and 1024 fails like this immediately on start:
vacuumdb: error: too many parallel jobs requested (maximum: 1023)

After 'ulimit -n 1200', vacuumdb -j1023 fails like this:

vacuumdb: vacuuming database "alvherre"
*** buffer overflow detected ***: vacuumdb terminated
Aborted

So I agree that we need a fix.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Commits

  1. Fix error handling of vacuumdb when running out of fds

  2. Fix error handling of vacuumdb and reindexdb when running out of fds

  3. Fix compilation failure of vacuumdb and reindexdb with OpenBSD

  4. Improve Assert output

  5. Refactor parallelization processing code in src/bin/scripts/

  6. vacuumdb: enable parallel mode