Add jsonlog log_destination for JSON server logs

Sehrope Sarkuni <sehrope@jackdb.com>

From: Sehrope Sarkuni <sehrope@jackdb.com>
To: Pg Hackers <pgsql-hackers@postgresql.org>
Cc: Michael Paquier <michael@paquier.xyz>, david@fetter.org
Date: 2021-08-31T15:34:56Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →
  1. Introduce log_destination=jsonlog

  2. Refactor set of routines specific to elog.c

  3. Fix compilation warning in syslogger.c

  4. Refactor per-destination file rotation in logging collector

  5. Refactor output file handling when forking syslogger under EXEC_BACKEND

  6. Add regression tests for csvlog with the logging collector

  7. Refactor the syslogger pipe protocol to use a bitmask for its options

Attachments

Hi,

This patch adds a new log_destination, "jsonlog", that writes log entries
as lines of JSON. It was originally started by David Fetter using
the jsonlog module by Michael Paquier (
https://github.com/michaelpq/pg_plugins/blob/master/jsonlog/jsonlog.c) as a
basis for how to serialize the log messages. Thanks to both of them because
this wouldn't be possible without that starting point.

The first commit splits out the destination in log pipe messages into its
own field. Previously it would piggyback on the "is_last" field. This adds
an int to the message size but makes the rest of the code easier to follow.

The second commit adds a TAP test for log_destination "csvlog". This was
done to both confirm that the previous change didn't break anything and as
a skeleton for the test in the next commit.

The third commit adds the new log_destination "jsonlog". The output format
is one line per entry with the top level output being a JSON object keyed
with the log fields. Newlines in the output fields are escaped as \n so the
output file has exactly one line per log entry. It also includes a new test
for verifying the JSON output with some basic regex checks (similar to the
csvlog test).

Here's a sample of what the log entries look like:

{"timestamp":"2021-08-31 10:15:25.129
EDT","user":"sehrope","dbname":"postgres","pid":12012,"remote_host":"[local]","session_id":"612e397d.2eec","line_num":1,"ps":"idle","session_start":"2021-08-31
10:15:25
EDT","vxid":"3/2","txid":"0","error_severity":"LOG","application_name":"
006_jsonlog.pl","message":"statement: SELECT 1/0"}

It builds and passes "make check-world" on Linux. It also includes code to
handle Windows as well but I have not actually tried building it there.

Regards,
-- Sehrope Sarkuni
Founder & CEO | JackDB, Inc. | https://www.jackdb.com/