Re: Add jsonlog log_destination for JSON server logs

Tom Lane <tgl@sss.pgh.pa.us>

From: Tom Lane <tgl@sss.pgh.pa.us>
To: Alvaro Herrera <alvherre@alvh.no-ip.org>
Cc: Michael Paquier <michael@paquier.xyz>, Andrew Dunstan <andrew@dunslane.net>, Andres Freund <andres@anarazel.de>, Sehrope Sarkuni <sehrope@jackdb.com>, Pg Hackers <pgsql-hackers@postgresql.org>, david@fetter.org
Date: 2022-02-11T01:18:56Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →
  1. Introduce log_destination=jsonlog

  2. Refactor set of routines specific to elog.c

  3. Fix compilation warning in syslogger.c

  4. Refactor per-destination file rotation in logging collector

  5. Refactor output file handling when forking syslogger under EXEC_BACKEND

  6. Add regression tests for csvlog with the logging collector

  7. Refactor the syslogger pipe protocol to use a bitmask for its options

Alvaro Herrera <alvherre@alvh.no-ip.org> writes:
> So, thinking about this, there is one important piece that is missing
> here, which is the ability to change the default format for what we
> write to stderr.  Right now, if you have stderr output, it is always in
> the "plain multiline" format, with no option to change it.  If you want
> a JSON log, you have to read a file.  But ISTM it would be pretty useful
> if you could say "log_default_format=json" and get the log that we get
> in stderr in the JSON format instead.

>> From what I hear in the container world, what they would *prefer* (but
> they don't often get) is to receive the JSON-format logs directly in
> stderr from the daemons they run; they capture stderr and they have the
> logs just in the format they need, without having to open the log files,
> parsing the lines to rewrite in a different format as is done currently.

> I think this would be a relatively easy patch to do.  Opinions?

I think assuming that everything that comes out on the postmaster's stderr
is generated by our code is hopelessly naive.  See for example glibc's
bleats when it detects malloc corruption, or when loading a shlib fails.
So I don't believe something like this can be made to work reliably.

The existing syslogger logic does have the ability to cope with
such out-of-protocol data.  So maybe, if you are using syslogger,
you could have it transform such messages into some
lowest-common-denominator jsonlog format.  But it's not going to
work to expect that to happen with raw stderr.

			regards, tom lane