Horton

Re: Add jsonlog log_destination for JSON server logs

Andrew Dunstan <andrew@dunslane.net>

View thread
From: Andrew Dunstan <andrew@dunslane.net>
To: Michael Paquier <michael@paquier.xyz>, Sehrope Sarkuni <sehrope@jackdb.com>
Cc: Pg Hackers <pgsql-hackers@postgresql.org>, david@fetter.org
Date: 2021-09-08T12:46:44Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Introduce log_destination=jsonlog

  2. Refactor set of routines specific to elog.c

  3. Fix compilation warning in syslogger.c

  4. Refactor per-destination file rotation in logging collector

  5. Refactor output file handling when forking syslogger under EXEC_BACKEND

  6. Add regression tests for csvlog with the logging collector

  7. Refactor the syslogger pipe protocol to use a bitmask for its options

On 9/8/21 2:58 AM, Michael Paquier wrote:
> On Wed, Sep 01, 2021 at 04:39:43PM -0400, Sehrope Sarkuni wrote:
>> That makes the elog.c changes in the JSON logging patch minimal as all it's
>> really doing is invoking the new write_jsonlog(...) function.
> Looking at 0001, to do things in order.
>
>> @@ -46,8 +46,8 @@ typedef struct
>>  	char		nuls[2];		/* always \0\0 */
>>  	uint16		len;			/* size of this chunk (counts data only) */
>>  	int32		pid;			/* writer's pid */
>> -	char		is_last;		/* last chunk of message? 't' or 'f' ('T' or
>> -								 * 'F' for CSV case) */
>> +	int32		dest;			/* log destination */
>> +	char		is_last;        /* last chunk of message? 't' or 'f'*/
>>  	char		data[FLEXIBLE_ARRAY_MEMBER];	/* data payload starts here */
>>  } PipeProtoHeader;
> Making PipeProtoHeader larger is not free, and that could penalize
> workloads with a lot of short messages and many backends as the
> syslogger relies on pipes with sync calls.  Why not switching is_last
> to bits8 flags instead?  That should be enough for the addition of
> JSON.  3 bits are enough at the end: one to know if it is the last
> chunk of message, one for CSV and one for JSON.



Yeah. A very simple change would be to use two different values for json
(say 'y' and 'n'). A slightly more principled scheme might use the top
bit for the end marker and the bottom 3 bits for the dest type (so up to
8 types possible), with the rest available for future use.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com