Horton

Thread

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. pg_createsubscriber: Don't use MAXPGPATH

  2. pg_createsubscriber: Remove separate logfile_open() function

  3. pg_createsubscriber: Use logging.c log file callback

  4. Add log file support to logging.c

  5. pg_createsubscriber: Add -l/--logdir option to redirect output to files.

  6. pg_createsubscriber: Introduce module-specific logging functions.

  1. [Proposal] Adding Log File Capability to pg_createsubscriber

    Gyan Sreejith <gyan.sreejith@gmail.com> — 2025-12-09T22:16:40Z

    Background:

   -

   pg_createsubscriber currently outputs all messages (internal validation
   messages, standby server start/stop logs, recovery progress output, and
   output from utilities) directly to the console. As a result, users may find
   debugging and handling errors difficult. It would be more convenient if
   messages were separated and stored in different log files. There is already
   a similar implementation in pg_upgrade.

Proposed Solution:

   -

   Based on issues mentioned previously, I would like to propose a new
   argument -l <logdir> which can be specified for pg_createsubscriber. Using
   it would create the following log files:
   -

      logdir/pg_createsubscriber_server.log which captures all logs related
      to starting and stopping the standby server.
      -

      logdir/pg_createsubscriber_resetwal.log which captures the output of
      pg_resetwal
      -

      logdir/pg_createsubscriber_internal.log which captures internal
      diagnostic output from pg_createsubscriber (validations, checks, etc.)

Overall, this proposed solution could make the pg_createsubscriber command
output messages more organized. The command would be easier to use as users
will only have to read individual log files rather than parse through lots
of possibly irrelevant output messages. I have attached the patch for this
change.

Special thanks to Vignesh C. for his offlist guidance on this project.


Regards, Gyan Sreejith

  2. Re: [Proposal] Adding Log File Capability to pg_createsubscriber

    Peter Smith <smithpb2250@gmail.com> — 2025-12-11T07:29:07Z

    On Wed, Dec 10, 2025 at 9:17 AM Gyan Sreejith <gyan.sreejith@gmail.com> wrote:
>
> Background:
>
> pg_createsubscriber currently outputs all messages (internal validation messages, standby server start/stop logs, recovery progress output, and output from utilities) directly to the console. As a result, users may find debugging and handling errors difficult. It would be more convenient if messages were separated and stored in different log files. There is already a similar implementation in pg_upgrade.
>
> Proposed Solution:
>
> Based on issues mentioned previously, I would like to propose a new argument -l <logdir> which can be specified for pg_createsubscriber. Using it would create the following log files:
>
> logdir/pg_createsubscriber_server.log which captures all logs related to starting and stopping the standby server.
>
> logdir/pg_createsubscriber_resetwal.log which captures the output of pg_resetwal
>
> logdir/pg_createsubscriber_internal.log which captures internal diagnostic output from pg_createsubscriber (validations, checks, etc.)
>
> Overall, this proposed solution could make the pg_createsubscriber command output messages more organized. The command would be easier to use as users will only have to read individual log files rather than parse through lots of possibly irrelevant output messages. I have attached the patch for this change.
>
> Special thanks to Vignesh C. for his offlist guidance on this project.
>
>
> Regards, Gyan Sreejith
>

Hi Gyan.

I haven't yet looked at this patch in any detail, but here are some
quick comments:

======

1.
+ printf(_("  -l, --logdir=LOGDIR             location for the new log
directory\n"));

The patch is missing SGML docs updates for pg_createsubscriber new
option, and any explanation of the split of logfiles.

2.
I might be mistaken, but IIUC it seems the splitting of the logfile
only works when --logdir is specified. Is that correct?
Why should --logdir have any side-effect other than assigning the log
destination folder?

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  3. Re: [Proposal] Adding Log File Capability to pg_createsubscriber

    Gyan Sreejith <gyan.sreejith@gmail.com> — 2025-12-12T01:33:15Z

    Thanks for the feedback, Peter.

I am currently working on the SGML docs update, and will promptly get back
with an update.

For your second point, currently, all output goes directly to the console.
I thought it made more sense to break it up into multiple files depending
on what was being invoked. Do you have another opinion?

Thank you once again,
Gyan Sreejith

On Thu, Dec 11, 2025 at 2:29 AM Peter Smith <smithpb2250@gmail.com> wrote:

> On Wed, Dec 10, 2025 at 9:17 AM Gyan Sreejith <gyan.sreejith@gmail.com>
> wrote:
> >
> > Background:
> >
> > pg_createsubscriber currently outputs all messages (internal validation
> messages, standby server start/stop logs, recovery progress output, and
> output from utilities) directly to the console. As a result, users may find
> debugging and handling errors difficult. It would be more convenient if
> messages were separated and stored in different log files. There is already
> a similar implementation in pg_upgrade.
> >
> > Proposed Solution:
> >
> > Based on issues mentioned previously, I would like to propose a new
> argument -l <logdir> which can be specified for pg_createsubscriber. Using
> it would create the following log files:
> >
> > logdir/pg_createsubscriber_server.log which captures all logs related to
> starting and stopping the standby server.
> >
> > logdir/pg_createsubscriber_resetwal.log which captures the output of
> pg_resetwal
> >
> > logdir/pg_createsubscriber_internal.log which captures internal
> diagnostic output from pg_createsubscriber (validations, checks, etc.)
> >
> > Overall, this proposed solution could make the pg_createsubscriber
> command output messages more organized. The command would be easier to use
> as users will only have to read individual log files rather than parse
> through lots of possibly irrelevant output messages. I have attached the
> patch for this change.
> >
> > Special thanks to Vignesh C. for his offlist guidance on this project.
> >
> >
> > Regards, Gyan Sreejith
> >
>
> Hi Gyan.
>
> I haven't yet looked at this patch in any detail, but here are some
> quick comments:
>
> ======
>
> 1.
> + printf(_("  -l, --logdir=LOGDIR             location for the new log
> directory\n"));
>
> The patch is missing SGML docs updates for pg_createsubscriber new
> option, and any explanation of the split of logfiles.
>
> 2.
> I might be mistaken, but IIUC it seems the splitting of the logfile
> only works when --logdir is specified. Is that correct?
> Why should --logdir have any side-effect other than assigning the log
> destination folder?
>
> ======
> Kind Regards,
> Peter Smith.
> Fujitsu Australia
>

  4. Re: [Proposal] Adding Log File Capability to pg_createsubscriber

    Gyan Sreejith <gyan.sreejith@gmail.com> — 2025-12-14T15:47:36Z

    I have included the patch file after making the changes to the SGML docs.

Thanks for your help,
Gyan Sreejith

On Thu, Dec 11, 2025 at 8:33 PM Gyan Sreejith <gyan.sreejith@gmail.com>
wrote:

> Thanks for the feedback, Peter.
>
> I am currently working on the SGML docs update, and will promptly get back
> with an update.
>
> For your second point, currently, all output goes directly to the console.
> I thought it made more sense to break it up into multiple files depending
> on what was being invoked. Do you have another opinion?
>
> Thank you once again,
> Gyan Sreejith
>
> On Thu, Dec 11, 2025 at 2:29 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
>> On Wed, Dec 10, 2025 at 9:17 AM Gyan Sreejith <gyan.sreejith@gmail.com>
>> wrote:
>> >
>> > Background:
>> >
>> > pg_createsubscriber currently outputs all messages (internal validation
>> messages, standby server start/stop logs, recovery progress output, and
>> output from utilities) directly to the console. As a result, users may find
>> debugging and handling errors difficult. It would be more convenient if
>> messages were separated and stored in different log files. There is already
>> a similar implementation in pg_upgrade.
>> >
>> > Proposed Solution:
>> >
>> > Based on issues mentioned previously, I would like to propose a new
>> argument -l <logdir> which can be specified for pg_createsubscriber. Using
>> it would create the following log files:
>> >
>> > logdir/pg_createsubscriber_server.log which captures all logs related
>> to starting and stopping the standby server.
>> >
>> > logdir/pg_createsubscriber_resetwal.log which captures the output of
>> pg_resetwal
>> >
>> > logdir/pg_createsubscriber_internal.log which captures internal
>> diagnostic output from pg_createsubscriber (validations, checks, etc.)
>> >
>> > Overall, this proposed solution could make the pg_createsubscriber
>> command output messages more organized. The command would be easier to use
>> as users will only have to read individual log files rather than parse
>> through lots of possibly irrelevant output messages. I have attached the
>> patch for this change.
>> >
>> > Special thanks to Vignesh C. for his offlist guidance on this project.
>> >
>> >
>> > Regards, Gyan Sreejith
>> >
>>
>> Hi Gyan.
>>
>> I haven't yet looked at this patch in any detail, but here are some
>> quick comments:
>>
>> ======
>>
>> 1.
>> + printf(_("  -l, --logdir=LOGDIR             location for the new log
>> directory\n"));
>>
>> The patch is missing SGML docs updates for pg_createsubscriber new
>> option, and any explanation of the split of logfiles.
>>
>> 2.
>> I might be mistaken, but IIUC it seems the splitting of the logfile
>> only works when --logdir is specified. Is that correct?
>> Why should --logdir have any side-effect other than assigning the log
>> destination folder?
>>
>> ======
>> Kind Regards,
>> Peter Smith.
>> Fujitsu Australia
>>
>

  5. RE: [Proposal] Adding Log File Capability to pg_createsubscriber

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2025-12-16T07:01:45Z

    Dear Gyan,

+1 for the idea. This point has already been discussed since the initial commit
[1], but it has left till now. I'm happy if you can take initiative.
Of course I can review your patches.

Per my understanding, pg_upgrade puts logfiles at the directory, under
"${PGDATANEW}/pg_upgrade_output.d/${timestamp}". See Note part in [2].
I feel more straightforward way is to follow that approach:

1. pg_createsubscriber creates a directory pg_createsubscriber_output.d/${timestamp}.
   ${timestamp} has the same format as ISO 8601 (%Y%m%dT%H%M%S).
2. pg_craetesubscriber saves outputs under the directory.
3. Outputs can be retained when the command failed or --retain is specified.
   Otherwise, they are removed at the end.

Are there benefits to provide -l option?

Regarding the patch format, our community prefers patches generated by
git format-patch. Can you see the blogpost [3] and try to create patches based on the command?
One benefit is we can easily do versioning.

[1]: https://www.postgresql.org/message-id/60b45b8a-3047-4a21-ba2a-ddb15daa638f%40eisentraut.org
[2]: https://www.postgresql.org/docs/devel/pgupgrade.html
[3]: https://peter.eisentraut.org/blog/2023/05/09/how-to-submit-a-patch-by-email-2023-edition

Best regards,
Hayato Kuroda
FUJITSU LIMITED

  6. Re: [Proposal] Adding Log File Capability to pg_createsubscriber

    Peter Smith <smithpb2250@gmail.com> — 2025-12-16T23:49:28Z

    On Fri, Dec 12, 2025 at 12:33 PM Gyan Sreejith <gyan.sreejith@gmail.com> wrote:
>
> Thanks for the feedback, Peter.
>
> I am currently working on the SGML docs update, and will promptly get back with an update.
>
> For your second point, currently, all output goes directly to the console. I thought it made more sense to break it up into multiple files depending on what was being invoked. Do you have another opinion?
>

The point I was trying to make was that if you are going to have a
"--logdir" option, then IMO that option should do nothing other than
change the destination for the logs. In other words, current behaviour
is effectively --logdir=console, so --logdir=some_other_folder should
not have some side-effect causing the log to get split into multiple
files.

If you want to split logs, then I thought Kuroda-San's suggestion [1]
sounded better ---  (a) mimic pg_upgrade more closely and (b)
reconsider if -logdir is needed at all.

======
[1] https://www.postgresql.org/message-id/OSCPR01MB14966FD0961F512B29BD46D6BF5AAA%40OSCPR01MB14966.jpnprd01.prod.outlook.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  7. Re: [Proposal] Adding Log File Capability to pg_createsubscriber

    vignesh C <vignesh21@gmail.com> — 2025-12-17T10:07:09Z

    On Tue, 16 Dec 2025 at 12:31, Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear Gyan,
>
> +1 for the idea. This point has already been discussed since the initial commit
> [1], but it has left till now. I'm happy if you can take initiative.
> Of course I can review your patches.
>
> Per my understanding, pg_upgrade puts logfiles at the directory, under
> "${PGDATANEW}/pg_upgrade_output.d/${timestamp}". See Note part in [2].
> I feel more straightforward way is to follow that approach:
>
> 1. pg_createsubscriber creates a directory pg_createsubscriber_output.d/${timestamp}.
>    ${timestamp} has the same format as ISO 8601 (%Y%m%dT%H%M%S).
> 2. pg_craetesubscriber saves outputs under the directory.
> 3. Outputs can be retained when the command failed or --retain is specified.
>    Otherwise, they are removed at the end.

If I recall correctly, this was implemented that way earlier, but the
approach was abandoned around [1]. The primary reason was that when
users take a backup of the data directory, they would need to
explicitly manage the exclusion of this data, which was considered
undesirable.

> Are there benefits to provide -l option?

By providing this as an option, users can store the log files outside
the data directory, eliminating the need for any additional handling
during backups.

[1] - https://www.postgresql.org/message-id/d546c4bb-92d1-4e2d-898f-48234b12ed25%40app.fastmail.com

Regards,
Vignesh

  8. Re: [Proposal] Adding Log File Capability to pg_createsubscriber

    Euler Taveira <euler@eulerto.com> — 2025-12-18T01:28:54Z

    On Wed, Dec 17, 2025, at 7:07 AM, vignesh C wrote:
>
> By providing this as an option, users can store the log files outside
> the data directory, eliminating the need for any additional handling
> during backups.
>

Do we really need an option to capture the stdout / stderr output to a file? I
doubt it. There is already various ways to capture. psql and pg_upgrade are the
only tools that have this option. And honestly, I've never seen these options
used in the field.


-- 
Euler Taveira
EDB   https://www.enterprisedb.com/

  9. Re: [Proposal] Adding Log File Capability to pg_createsubscriber

    Amit Kapila <amit.kapila16@gmail.com> — 2025-12-18T06:49:02Z

    On Thu, Dec 18, 2025 at 6:59 AM Euler Taveira <euler@eulerto.com> wrote:
>
> On Wed, Dec 17, 2025, at 7:07 AM, vignesh C wrote:
> >
> > By providing this as an option, users can store the log files outside
> > the data directory, eliminating the need for any additional handling
> > during backups.
> >
>
> Do we really need an option to capture the stdout / stderr output to a file? I
> doubt it. There is already various ways to capture. psql and pg_upgrade are the
> only tools that have this option.
>

pg_ctl also has the -l option. I think any place where long
text/errors can be outputted, a log file is preferred because one
could later parse it to know the exact details. Also, splitting the
log as proposed here or in pg_upgrade helps to navigate the LOG like
is the problem in start/stop of the server or a pub-sub setup?
Similarly the log can be splitted for pub/sub specific information.
There appears to be some useful information like:

pg_createsubscriber: warning: two_phase option will not be enabled for
replication slots
pg_createsubscriber: detail: Subscriptions will be created with the
two_phase option disabled. Prepared transactions will be replicated at
COMMIT PREPARED.
pg_createsubscriber: hint: You can use the command-line option
--enable-two-phase to enable two_phase.

I think it will be useful to LOG this separately from the main LOG [1]
(which can contain server specific info as follows) so that users can
consider running pg_createsubscriber with additional options or
changing the subscriber configuration once setup is complete.

[1]:
[startup] LOG:  database system was interrupted; last known up at
2025-12-17 14:46:07 IST
[startup] LOG:  starting backup recovery with redo LSN 0/06000028,
checkpoint LSN 0/06000080, on timeline ID 1
[startup] LOG:  entering standby mode
[startup] LOG:  redo starts at 0/06000028
[startup] LOG:  completed backup recovery with redo LSN 0/06000028 and
end LSN 0/06000120
[startup] LOG:  consistent recovery state reached at 0/06000120

-- 
With Regards,
Amit Kapila.

  10. Re: [Proposal] Adding Log File Capability to pg_createsubscriber

    Gyan Sreejith <gyan.sreejith@gmail.com> — 2025-12-23T23:22:11Z

    Thank you for the feedback everybody. As I read through this email chain, I
found differing opinions on how logging should be implemented. This
ambiguity leaves me unsure as to which solution(s) to pursue. As of right
now, I have attached the git-format patch like Hayato Kuroda recommended
(but it does not have any new changes). I am willing to implement whatever
solution when we reach a consensus.

Thank you for all of the help,
Gyan Sreejith

On Thu, Dec 18, 2025 at 1:49 AM Amit Kapila <amit.kapila16@gmail.com> wrote:

> On Thu, Dec 18, 2025 at 6:59 AM Euler Taveira <euler@eulerto.com> wrote:
> >
> > On Wed, Dec 17, 2025, at 7:07 AM, vignesh C wrote:
> > >
> > > By providing this as an option, users can store the log files outside
> > > the data directory, eliminating the need for any additional handling
> > > during backups.
> > >
> >
> > Do we really need an option to capture the stdout / stderr output to a
> file? I
> > doubt it. There is already various ways to capture. psql and pg_upgrade
> are the
> > only tools that have this option.
> >
>
> pg_ctl also has the -l option. I think any place where long
> text/errors can be outputted, a log file is preferred because one
> could later parse it to know the exact details. Also, splitting the
> log as proposed here or in pg_upgrade helps to navigate the LOG like
> is the problem in start/stop of the server or a pub-sub setup?
> Similarly the log can be splitted for pub/sub specific information.
> There appears to be some useful information like:
>
> pg_createsubscriber: warning: two_phase option will not be enabled for
> replication slots
> pg_createsubscriber: detail: Subscriptions will be created with the
> two_phase option disabled. Prepared transactions will be replicated at
> COMMIT PREPARED.
> pg_createsubscriber: hint: You can use the command-line option
> --enable-two-phase to enable two_phase.
>
> I think it will be useful to LOG this separately from the main LOG [1]
> (which can contain server specific info as follows) so that users can
> consider running pg_createsubscriber with additional options or
> changing the subscriber configuration once setup is complete.
>
> [1]:
> [startup] LOG:  database system was interrupted; last known up at
> 2025-12-17 14:46:07 IST
> [startup] LOG:  starting backup recovery with redo LSN 0/06000028,
> checkpoint LSN 0/06000080, on timeline ID 1
> [startup] LOG:  entering standby mode
> [startup] LOG:  redo starts at 0/06000028
> [startup] LOG:  completed backup recovery with redo LSN 0/06000028 and
> end LSN 0/06000120
> [startup] LOG:  consistent recovery state reached at 0/06000120
>
> --
> With Regards,
> Amit Kapila.
>

  11. Re: [Proposal] Adding Log File Capability to pg_createsubscriber

    vignesh C <vignesh21@gmail.com> — 2025-12-29T11:10:46Z

    On Wed, 24 Dec 2025 at 04:52, Gyan Sreejith <gyan.sreejith@gmail.com> wrote:
>
> Thank you for the feedback everybody. As I read through this email chain, I found differing opinions on how logging should be implemented. This ambiguity leaves me unsure as to which solution(s) to pursue. As of right now, I have attached the git-format patch like Hayato Kuroda recommended (but it does not have any new changes). I am willing to implement whatever solution when we reach a consensus.

Few comments:
1) The file permissions are 664 for pg_createsubscriber_internal.log,
pg_createsubscriber_resetwal.log but 600 for
pg_createsubscriber_server.log. The permissions should be the same for
all the files.
...
if (opt->log_dir != NULL)
out_file = psprintf("%s/pg_createsubscriber_resetwal.log", opt->log_dir);
else
out_file = DEVNULL;

cmd_str = psprintf("\"%s\" -D \"%s\" > \"%s\"", pg_resetwal_path,
   subscriber_dir, out_file);

pg_log_debug("pg_resetwal command is: %s", cmd_str);
...

...
if (opt->log_dir != NULL)
{
appendPQExpBuffer(pg_ctl_cmd, " -l %s/pg_createsubscriber_server.log",
opt->log_dir);
}

pg_log_debug("pg_ctl command is: %s", pg_ctl_cmd->data);
rc = system(pg_ctl_cmd->data);
...
2)  Can you gracefully handle the case where permissions are not
enough in the directory and throw proper error:
if (stat(opt.log_dir, &statbuf) != 0)
{
if (errno == ENOENT)
{
mkdir(opt.log_dir, S_IRWXU);
pg_log_info("log directory created");
}
else
pg_fatal("could not access directory \"%s\": %m", opt.log_dir);
}

3) Currently there is no timestamp included for
pg_createsubscriber_internal and pg_createsubscriber_resetwal log file
contents. Without that it is difficult to tell when the operations
were done. It will be good to include them.

4) The patch does not apply on the head, kindly rebase on top of head.

5) Do you need to open and close the log file each time?
...
if (internal_log_file != NULL)
{
if ((fp = fopen(internal_log_file, "a")) == NULL)
pg_fatal("could not write to log file \"%s\": %m", internal_log_file);

fprintf(fp, "checking settings on subscriber\n");
fclose(fp);
}
else
pg_log_info("checking settings on subscriber");
...

...
if (internal_log_file != NULL)
{
if ((fp = fopen(internal_log_file, "a")) == NULL)
pg_fatal("could not write to log file \"%s\": %m", internal_log_file);

fprintf(fp, "checking settings on publisher\n");
fclose(fp);
}
else
pg_log_info("checking settings on publisher");
...

Regards,
Vignesh