Thread

1. BUG #18856: Include directives for postgresql.conf, pg_hba.conf, pg_ident.conf behave inconsistently.

   PG Bug reporting form <noreply@postgresql.org> — 2025-03-18T22:30:08Z

   The following bug has been logged on the website:
   
   Bug reference:      18856
   Logged by:          Robert Pufky
   Email address:      rpufky@gmail.com
   PostgreSQL version: 17.4
   Operating system:   Debian Bookworm
   Description:        
   
   This is a nuanced error; please read the entire report. Only changed lines
   are shown.
   
   Effectively, the include_dir, include_if_exists, and include directives
   behave differently for each of the files, even though there is no
   documentation stating any deviation in behavior. My working assumption is
   that the behavior in postgresql.conf for those directives carries over to
   pg_hba.conf and pg_ident.conf; which it does not.
   
   Complete setup and testing cases below validating what I am seeing:
   
   System Configuration:
   * debian bookworm, latest patches, all repositories; 2025-03-18
   * Postgres 17.4, from postgres repos - https://apt.postgres.org/;
   2025-03-18
   * Default package installation (apt update && apt dist-upgrade && apt
   install postgres)
   
   /tmp/pg.conf:
   max_connections=1000
   
   /tmp/pg_ident.conf
   local_all postgres trust
   /tmp/pg_hba.conf
   local all all trust
   
   Test: postgresql.conf
   * Quoting sources work correctly. This is expected based on documentation.
   
   .../17/main/postgresql.conf:
   include_dir = 'conf.d'
   include_if_exists = '/tmp/pg.conf'
   
   -> postgresql service starts. No errors. Expected.
   ---
   .../17/main/postgresql.conf:
   include_dir 'conf.d'
   include_if_exists '/tmp/pg.conf'
   
   -> postgresql service starts. No errors. Expected.
   
   Test: postgresql.conf unquoted
   * Not quoting sources work correctly. This is expected based on
   documentation.
   
   .../17/main/postgresql.conf:
   include_dir = conf.d
   include_if_exists = /tmp/pg.conf
   
   -> postgresql service fails to start. Expected.
   2025-03-18 21:32:09.317 GMT [8970] LOG:  syntax error in file
   "/etc/postgresql/17/main/postgresql.conf" line 834, near token "conf.d"
   2025-03-18 21:32:09.317 GMT [8970] LOG:  syntax error in file
   "/etc/postgresql/17/main/postgresql.conf" line 836, near token "/"
   2025-03-18 21:32:09.317 GMT [8970] FATAL:  configuration file
   "/etc/postgresql/17/main/postgresql.conf" contains errors
   ---
   .../17/main/postgresql.conf:
   include_dir conf.d
   include_if_exists /tmp/pg.conf
   
   -> postgresql service fails to start. Expected.
   2025-03-18 21:32:09.317 GMT [8970] LOG:  syntax error in file
   "/etc/postgresql/17/main/postgresql.conf" line 834, near token "conf.d"
   2025-03-18 21:32:09.317 GMT [8970] LOG:  syntax error in file
   "/etc/postgresql/17/main/postgresql.conf" line 836, near token "/"
   2025-03-18 21:32:09.317 GMT [8970] FATAL:  configuration file
   "/etc/postgresql/17/main/postgresql.conf" contains errors
   
   
   Test: pg_ident.conf
   * Expectation: Quoting sources as before works as-in postgresql.conf;
   per pg_ident.conf documentation and include documentation
   
   .../17/main/pg_hba.conf
   local all all peer map=local_all
   
   .../17/main/pg_ident.conf
   include_dir = 'conf.d'
   include_if_exists = '/tmp/pg_ident.conf'
   
   -> postgresql service starts. No errors. Expected.
   ---
   .../17/main/pg_hba.conf
   local all all peer map=local_all
   
   .../17/main/pg_ident.conf
   include_dir = conf.d
   include_if_exists = /tmp/pg_ident.conf
   
   -> postgresql service starts. No errors. DEVIATION from postgresql.conf
   behavior.
   ---
   .../17/main/pg_hba.conf
   local all all peer map=local_all
   
   .../17/main/pg_ident.conf
   include_dir conf.d
   include_if_exists tmp/pg_ident.conf
   
   -> postgresql service starts. No errors. DEVIATION from postgresql.conf
   behavior.
   
   
   Test: pg_hba.conf
   * Expectation: Quoting sources as before works as-in postgresql.conf;
   per pg_hba.conf documentation and include documentation
   
   .../17/main/pg_hba.conf
   include_dir = 'conf.d'
   include_if_exists = '/tmp/pg_hba.conf'
   
   -> postgresql service fails to start. DEVIATION from postgresql.conf
   behavior.
   2025-03-18 22:04:47.437 UTC [9110] LOG:  invalid connection type
   "include_dir"
   2025-03-18 22:04:47.437 UTC [9110] CONTEXT:  line 134 of configuration file
   "/etc/postgresql/17/main/pg_hba.conf"
   2025-03-18 22:04:47.437 UTC [9110] LOG:  invalid connection type
   "include_if_exists"
   2025-03-18 22:04:47.437 UTC [9110] CONTEXT:  line 135 of configuration file
   "/etc/postgresql/17/main/pg_hba.conf"
   2025-03-18 22:04:47.437 UTC [9110] FATAL:  could not load
   /etc/postgresql/17/main/pg_hba.conf
   * include_if_exists explicitly supported in documentation.
   ---
   .../17/main/pg_hba.conf
   local all all peer map=local_all
   
   .../17/main/pg_ident.conf
   include_dir = conf.d
   include_if_exists = /tmp/pg_hba.conf
   
   -> postgresql service fails to start. DEVIATION from postgresql.conf
   behavior.
   2025-03-18 22:01:04.933 UTC [9109] LOG:  invalid connection type
   "include_dir"
   2025-03-18 22:01:04.933 UTC [9109] CONTEXT:  line 134 of configuration file
   "/etc/postgresql/17/main/pg_hba.conf"
   2025-03-18 22:01:04.933 UTC [9109] LOG:  invalid connection type
   "include_if_exists"
   2025-03-18 22:01:04.933 UTC [9109] CONTEXT:  line 135 of configuration file
   "/etc/postgresql/17/main/pg_hba.conf"
   2025-03-18 22:01:04.933 UTC [9109] FATAL:  could not load
   /etc/postgresql/17/main/pg_hba.conf
   * include_dir, include_if_exists explicitly supported in documentation.
   ---
   .../17/main/pg_hba.conf
   local all all peer map=local_all
   
   .../17/main/pg_ident.conf
   include_dir 'conf.d'
   include_if_exists '/tmp/pg_hba.conf'
   
   -> postgresql service fails to start. DEVIATION from postgresql.conf
   behavior.
   2025-03-18 22:05:26.253 UTC [9111] LOG:  could not open configuration
   directory "/etc/postgresql/17/main/'conf.d'": No such file or directory
   2025-03-18 22:05:26.253 UTC [9111] CONTEXT:  line 134 of configuration file
   "/etc/postgresql/17/main/pg_hba.conf"
   2025-03-18 22:05:26.253 UTC [9111] LOG:  could not open file
   "/etc/postgresql/17/main/'/tmp/pg_hba.conf'": No such file or directory
   2025-03-18 22:05:26.253 UTC [9111] CONTEXT:  line 135 of configuration file
   "/etc/postgresql/17/main/pg_hba.conf"
   2025-03-18 22:05:26.253 UTC [9111] LOG:  skipping missing authentication
   file "/etc/postgresql/17/main/'/tmp/pg_hba.conf'"
   2025-03-18 22:05:26.253 UTC [9111] CONTEXT:  line 135 of configuration file
   "/etc/postgresql/17/main/pg_hba.conf"
   2025-03-18 22:05:26.253 UTC [9111] FATAL:  could not load
   /etc/postgresql/17/main/pg_hba.conf
   * Looks like this now directly appends any quoted option directly to the
   data_directory path, including quotes, no path resolution.
   ---
   .../17/main/pg_hba.conf
   local all all peer map=local_all
   
   .../17/main/pg_ident.conf
   include_dir conf.d
   include_if_exists /tmp/pg_hba.conf
   
   -> postgresql service starts. DEVIATION from postgresql.conf behavior.
   select * from pg_hba_file_rules where file_name='/tmp/hba.conf';
    rule_number |    file_name     | line_number | type  | database | user_name
   | address | netmask | auth_method | options | error
   -------------+------------------+-------------+-------+----------+-----------+---------+---------+-------
   ------+---------+-------
              8 | /tmp/pg_hba.conf |           1 | local | {all}    | {all}    
   |         |         | trust
         |         |
   (1 row)
   * Looks like this loaded everything correctly?
   
   I believe this is an actual bug, or documentation needs to be explicitly
   updated to reflect behavior changes for include directives.
   
   include documentation source:
   https://www.postgresql.org/docs/17/config-setting.html#CONFIG-INCLUDES
   pg_ident.conf documentation source:
   https://www.postgresql.org/docs/17/auth-username-maps.html
   pg_hba.conf documentation source:
   https://www.postgresql.org/docs/current/auth-pg-hba-conf.html
   
   Thanks!
   
   

2. Re: BUG #18856: Include directives for postgresql.conf, pg_hba.conf, pg_ident.conf behave inconsistently.

   Robert Pufky <rpufky@gmail.com> — 2025-03-18T22:38:06Z

   For the 'Test: pg_hba.conf' tests, I included the wrong testing notes, they
   should be:
   
   Test 2:
   .../17/main/pg_hba.conf
   include_dir = conf.d
   include_if_exists = /tmp/pg_hba.conf
   
   Test 3:
   .../17/main/pg_hba.conf
   include_dir 'conf.d'
   include_if_exists '/tmp/pg_hba.conf'
   
   Test 4:
   .../17/main/pg_hba.conf
   include_dir conf.d
   include_if_exists /tmp/pg_hba.conf
   
   On Tue, Mar 18, 2025 at 3:31 PM PG Bug reporting form <
   noreply@postgresql.org> wrote:
   
   > The following bug has been logged on the website:
   >
   > Bug reference:      18856
   > Logged by:          Robert Pufky
   > Email address:      rpufky@gmail.com
   > PostgreSQL version: 17.4
   > Operating system:   Debian Bookworm
   > Description:
   >
   > This is a nuanced error; please read the entire report. Only changed lines
   > are shown.
   >
   > Effectively, the include_dir, include_if_exists, and include directives
   > behave differently for each of the files, even though there is no
   > documentation stating any deviation in behavior. My working assumption is
   > that the behavior in postgresql.conf for those directives carries over to
   > pg_hba.conf and pg_ident.conf; which it does not.
   >
   > Complete setup and testing cases below validating what I am seeing:
   >
   > System Configuration:
   > * debian bookworm, latest patches, all repositories; 2025-03-18
   > * Postgres 17.4, from postgres repos - https://apt.postgres.org/;
   > 2025-03-18
   > * Default package installation (apt update && apt dist-upgrade && apt
   > install postgres)
   >
   > /tmp/pg.conf:
   > max_connections=1000
   >
   > /tmp/pg_ident.conf
   > local_all postgres trust
   > /tmp/pg_hba.conf
   > local all all trust
   >
   > Test: postgresql.conf
   > * Quoting sources work correctly. This is expected based on documentation.
   >
   > .../17/main/postgresql.conf:
   > include_dir = 'conf.d'
   > include_if_exists = '/tmp/pg.conf'
   >
   > -> postgresql service starts. No errors. Expected.
   > ---
   > .../17/main/postgresql.conf:
   > include_dir 'conf.d'
   > include_if_exists '/tmp/pg.conf'
   >
   > -> postgresql service starts. No errors. Expected.
   >
   > Test: postgresql.conf unquoted
   > * Not quoting sources work correctly. This is expected based on
   > documentation.
   >
   > .../17/main/postgresql.conf:
   > include_dir = conf.d
   > include_if_exists = /tmp/pg.conf
   >
   > -> postgresql service fails to start. Expected.
   > 2025-03-18 21:32:09.317 GMT [8970] LOG:  syntax error in file
   > "/etc/postgresql/17/main/postgresql.conf" line 834, near token "conf.d"
   > 2025-03-18 21:32:09.317 GMT [8970] LOG:  syntax error in file
   > "/etc/postgresql/17/main/postgresql.conf" line 836, near token "/"
   > 2025-03-18 21:32:09.317 GMT [8970] FATAL:  configuration file
   > "/etc/postgresql/17/main/postgresql.conf" contains errors
   > ---
   > .../17/main/postgresql.conf:
   > include_dir conf.d
   > include_if_exists /tmp/pg.conf
   >
   > -> postgresql service fails to start. Expected.
   > 2025-03-18 21:32:09.317 GMT [8970] LOG:  syntax error in file
   > "/etc/postgresql/17/main/postgresql.conf" line 834, near token "conf.d"
   > 2025-03-18 21:32:09.317 GMT [8970] LOG:  syntax error in file
   > "/etc/postgresql/17/main/postgresql.conf" line 836, near token "/"
   > 2025-03-18 21:32:09.317 GMT [8970] FATAL:  configuration file
   > "/etc/postgresql/17/main/postgresql.conf" contains errors
   >
   >
   > Test: pg_ident.conf
   > * Expectation: Quoting sources as before works as-in postgresql.conf;
   > per pg_ident.conf documentation and include documentation
   >
   > .../17/main/pg_hba.conf
   > local all all peer map=local_all
   >
   > .../17/main/pg_ident.conf
   > include_dir = 'conf.d'
   > include_if_exists = '/tmp/pg_ident.conf'
   >
   > -> postgresql service starts. No errors. Expected.
   > ---
   > .../17/main/pg_hba.conf
   > local all all peer map=local_all
   >
   > .../17/main/pg_ident.conf
   > include_dir = conf.d
   > include_if_exists = /tmp/pg_ident.conf
   >
   > -> postgresql service starts. No errors. DEVIATION from postgresql.conf
   > behavior.
   > ---
   > .../17/main/pg_hba.conf
   > local all all peer map=local_all
   >
   > .../17/main/pg_ident.conf
   > include_dir conf.d
   > include_if_exists tmp/pg_ident.conf
   >
   > -> postgresql service starts. No errors. DEVIATION from postgresql.conf
   > behavior.
   >
   >
   > Test: pg_hba.conf
   > * Expectation: Quoting sources as before works as-in postgresql.conf;
   > per pg_hba.conf documentation and include documentation
   >
   > .../17/main/pg_hba.conf
   > include_dir = 'conf.d'
   > include_if_exists = '/tmp/pg_hba.conf'
   >
   > -> postgresql service fails to start. DEVIATION from postgresql.conf
   > behavior.
   > 2025-03-18 22:04:47.437 UTC [9110] LOG:  invalid connection type
   > "include_dir"
   > 2025-03-18 22:04:47.437 UTC [9110] CONTEXT:  line 134 of configuration file
   > "/etc/postgresql/17/main/pg_hba.conf"
   > 2025-03-18 22:04:47.437 UTC [9110] LOG:  invalid connection type
   > "include_if_exists"
   > 2025-03-18 22:04:47.437 UTC [9110] CONTEXT:  line 135 of configuration file
   > "/etc/postgresql/17/main/pg_hba.conf"
   > 2025-03-18 22:04:47.437 UTC [9110] FATAL:  could not load
   > /etc/postgresql/17/main/pg_hba.conf
   > * include_if_exists explicitly supported in documentation.
   > ---
   > .../17/main/pg_hba.conf
   > local all all peer map=local_all
   >
   > .../17/main/pg_ident.conf
   > include_dir = conf.d
   > include_if_exists = /tmp/pg_hba.conf
   >
   > -> postgresql service fails to start. DEVIATION from postgresql.conf
   > behavior.
   > 2025-03-18 22:01:04.933 UTC [9109] LOG:  invalid connection type
   > "include_dir"
   > 2025-03-18 22:01:04.933 UTC [9109] CONTEXT:  line 134 of configuration file
   > "/etc/postgresql/17/main/pg_hba.conf"
   > 2025-03-18 22:01:04.933 UTC [9109] LOG:  invalid connection type
   > "include_if_exists"
   > 2025-03-18 22:01:04.933 UTC [9109] CONTEXT:  line 135 of configuration file
   > "/etc/postgresql/17/main/pg_hba.conf"
   > 2025-03-18 22:01:04.933 UTC [9109] FATAL:  could not load
   > /etc/postgresql/17/main/pg_hba.conf
   > * include_dir, include_if_exists explicitly supported in documentation.
   > ---
   > .../17/main/pg_hba.conf
   > local all all peer map=local_all
   >
   > .../17/main/pg_ident.conf
   > include_dir 'conf.d'
   > include_if_exists '/tmp/pg_hba.conf'
   >
   > -> postgresql service fails to start. DEVIATION from postgresql.conf
   > behavior.
   > 2025-03-18 22:05:26.253 UTC [9111] LOG:  could not open configuration
   > directory "/etc/postgresql/17/main/'conf.d'": No such file or directory
   > 2025-03-18 22:05:26.253 UTC [9111] CONTEXT:  line 134 of configuration file
   > "/etc/postgresql/17/main/pg_hba.conf"
   > 2025-03-18 22:05:26.253 UTC [9111] LOG:  could not open file
   > "/etc/postgresql/17/main/'/tmp/pg_hba.conf'": No such file or directory
   > 2025-03-18 22:05:26.253 UTC [9111] CONTEXT:  line 135 of configuration file
   > "/etc/postgresql/17/main/pg_hba.conf"
   > 2025-03-18 22:05:26.253 UTC [9111] LOG:  skipping missing authentication
   > file "/etc/postgresql/17/main/'/tmp/pg_hba.conf'"
   > 2025-03-18 22:05:26.253 UTC [9111] CONTEXT:  line 135 of configuration file
   > "/etc/postgresql/17/main/pg_hba.conf"
   > 2025-03-18 22:05:26.253 UTC [9111] FATAL:  could not load
   > /etc/postgresql/17/main/pg_hba.conf
   > * Looks like this now directly appends any quoted option directly to the
   > data_directory path, including quotes, no path resolution.
   > ---
   > .../17/main/pg_hba.conf
   > local all all peer map=local_all
   >
   > .../17/main/pg_ident.conf
   > include_dir conf.d
   > include_if_exists /tmp/pg_hba.conf
   >
   > -> postgresql service starts. DEVIATION from postgresql.conf behavior.
   > select * from pg_hba_file_rules where file_name='/tmp/hba.conf';
   >  rule_number |    file_name     | line_number | type  | database |
   > user_name
   > | address | netmask | auth_method | options | error
   >
   > -------------+------------------+-------------+-------+----------+-----------+---------+---------+-------
   > ------+---------+-------
   >            8 | /tmp/pg_hba.conf |           1 | local | {all}    | {all}
   >
   > |         |         | trust
   >       |         |
   > (1 row)
   > * Looks like this loaded everything correctly?
   >
   > I believe this is an actual bug, or documentation needs to be explicitly
   > updated to reflect behavior changes for include directives.
   >
   > include documentation source:
   > https://www.postgresql.org/docs/17/config-setting.html#CONFIG-INCLUDES
   > pg_ident.conf documentation source:
   > https://www.postgresql.org/docs/17/auth-username-maps.html
   > pg_hba.conf documentation source:
   > https://www.postgresql.org/docs/current/auth-pg-hba-conf.html
   >
   > Thanks!
   >
   >
   

3. Re: BUG #18856: Include directives for postgresql.conf, pg_hba.conf, pg_ident.conf behave inconsistently.

   Michael Paquier <michael@paquier.xyz> — 2025-03-28T01:24:21Z

   On Tue, Mar 18, 2025 at 03:38:06PM -0700, Robert Pufky wrote:
   > For the 'Test: pg_hba.conf' tests, I included the wrong testing notes, they
   > should be:
   > 
   > Test 2:
   > .../17/main/pg_hba.conf
   > include_dir = conf.d
   > include_if_exists = /tmp/pg_hba.conf
   > 
   > Test 3:
   > .../17/main/pg_hba.conf
   > include_dir 'conf.d'
   > include_if_exists '/tmp/pg_hba.conf'
   > 
   > Test 4:
   > .../17/main/pg_hba.conf
   > include_dir conf.d
   > include_if_exists /tmp/pg_hba.conf
   
   The logic used for configuration files regarding GUCs (aka
   postgresql.conf) and HBA/ident files are bound to different rules when
   it comes to quotes.  For the HBA/ident files, the logic is around
   next_token() in hba.c.  For GUC files, the logic is much smarter, see
   guc-file.l.  The handling of the quotes and separators is different
   based on these rules, so, while we have a consistent set of grammar
   keywords to use for both, you may see differences like the ones you
   are reporting here.
   
   Your confusion here is behind the use of the "=" parameter with one of
   the include directives, making the HBA/ident code parsing think that
   this is a connection option, but well, it's not.   The documentation
   does not mention that this combination is OK:
   https://www.postgresql.org/docs/devel/auth-pg-hba-conf.html 
   https://www.postgresql.org/docs/devel/auth-username-maps.html
   
   Perhaps you're right and we should expand this error message to
   provide more details, or make the token parsing slightly smarter in
   this case by feeding optional "=" separators.  However, I am not
   convinced that this is worth the effort just on the ground to make
   HBA/ident parsing closer to the GUC parsing, because they are entirely
   different facilities, and the docs clearly state that '=' is not part
   of the supported grammar for the include keywords in the HBA and ident
   parts.  Hence, the HBA parsing generating this "invalid connection
   type" error is not completely wrong, because it's telling what it sees
   based on the tokens parsed.
   --
   Michael