Horton

Re: Add jsonlog log_destination for JSON server logs

Sehrope Sarkuni <sehrope@jackdb.com>

View thread
From: Sehrope Sarkuni <sehrope@jackdb.com>
To: Michael Paquier <michael@paquier.xyz>
Cc: Pg Hackers <pgsql-hackers@postgresql.org>, david@fetter.org
Date: 2021-09-01T12:33:54Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Introduce log_destination=jsonlog

  2. Refactor set of routines specific to elog.c

  3. Fix compilation warning in syslogger.c

  4. Refactor per-destination file rotation in logging collector

  5. Refactor output file handling when forking syslogger under EXEC_BACKEND

  6. Add regression tests for csvlog with the logging collector

  7. Refactor the syslogger pipe protocol to use a bitmask for its options

On Tue, Aug 31, 2021 at 8:43 PM Michael Paquier <michael@paquier.xyz> wrote:

> On Tue, Aug 31, 2021 at 11:34:56AM -0400, Sehrope Sarkuni wrote:
> > The second commit adds a TAP test for log_destination "csvlog". This was
> > done to both confirm that the previous change didn't break anything and
> as
> > a skeleton for the test in the next commit.
>
> +note "Before sleep";
> +usleep(100_000);
> +note "Before rotate";
> +$node->logrotate();
> +note "After rotate";
> +usleep(100_000);
>
> Do you really need a rotation of the log files here?  Wouldn't it be
> better to grab the position of the current log file with a fixed log
> file name, and then slurp the file from this position with your
> expected output?  That would make the test faster, as well.
>

Yes, that was intentional. I used the log rotation tap test as a base and
kept that piece in there so it verifies that the csv log files are actually
rotated. Same for the json logs.


> Rather than making elog.c larger, I think that we should try to split
> that into more files.  Why not refactoring out the CSV part first?
> You could just call that csvlog.c, then create a new jsonlog.c for the
> meat of the patch.
>

That's a good idea. I'll try that out.

The list of fields is not up to date.  At quick glance, you are
> missing:
> - backend type.
> - leader PID.
> - query ID.
> - Session start timestamp (?)
>

Thanks. I'll take a look.

Regards,
-- Sehrope Sarkuni
Founder & CEO | JackDB, Inc. | https://www.jackdb.com/