Thread

1. Sketch of a Hook into the Logging Collector

   Daniel Farina <daniel@heroku.com> — 2012-12-08T18:40:12Z

   Hello all,
   
   I have some needs that seem to support changing Postgres slightly to
   give user programs a lot more power over how to process logging output
   that neither the log collector nor the syslog output can well-satisfy
   as-is.
   
   I am approaching this from the angle of increasing power by exposing
   the log collector ("syslogger") pipe protocol.
   
   Included is a patch whose general aesthetic is to touch as little of
   the logging code as possible while achieving maximum power in a
   contrib and stand-alone sample program.  However, a more satisfying
   treatment that may be slightly more invasive to the logging system is
   very welcome.
   
   The general idea here is that the logging collector pipe protocol is a
   pretty reasonable one for non-Postgres programs to consume as-is
   pretty easily.  The proof of concept consists of three things:
   
   * A hook into the log collector.  It is small but a little weird
     compared to the other hooks in the system.
   
   * A contrib, pg_logcollectdup.  This contrib lets one forward logs to
     a named pipe specified in postgresql.conf.
   
   * A stand-alone program, pg_logcollect_sample, that renders protocol
     traffic newline separated and with its binary fields converted to
     readable text.
   
   The patch can also be seen from https://github.com/fdr/postgres.git in
   the branch 'logdup'.  I do rebase this at-will for the time being.
   
   I have a few detailed dissatisfactions with this approach, but I'd
   rather hear your own dissatisfactions.  I also don't like the name.
   
   A demo of configuring all of the above and seeing some output follows.
   I can paste this in one go on my machine.  It creates /tmp/pg-logdup
   and /tmp/pgdata-logdup.  All spawned processes can be listed
   afterwards with 'jobs'.
   
   # Install postgres and pg_logcollectdup contrib
   ./configure --prefix=/tmp/pg-logdup
   make -sj16 install
   cd contrib/pg_logcollectdup/
   make -s install
   /tmp/pg-logdup/bin/initdb -D /tmp/pgdata-logdup
   mkfifo /tmp/pgdata-logdup/log-pipe
   
   # Configure postgresql.conf
   
   echo "logging_collector = on" >> \
    /tmp/pgdata-logdup/postgresql.conf
   
   echo "shared_preload_libraries = 'pg_logcollectdup'" >> \
   /tmp/pgdata-logdup/postgresql.conf
   
   echo "logcollectdup.destination = '/tmp/pgdata-logdup/log-pipe'" >> \
   /tmp/pgdata-logdup/postgresql.conf
   
   # Set up destination pipe
   mkfifo /tmp/pgdata-logdup/log-pipe
   
   # Build sample pipe formatter
   make pg_logcollect_sample
   
   # Run it in the background
   ./pg_logcollect_sample            \
   < /tmp/pgdata-logdup/log-pipe \
   > /tmp/pgdata-logdup/duplogs.txt &
   
   # Run Postgres with a non-default port for convenience
   /tmp/pg-logdup/bin/postgres -D /tmp/pgdata-logdup --port=2345 &
   
   # Prevent race against file creation, then look at the logs
   sleep 1
   cat /tmp/pgdata-logdup/duplogs.txt
   
   --
   fdr
   

2. Re: Sketch of a Hook into the Logging Collector

   Daniel Farina <daniel@heroku.com> — 2012-12-08T19:36:15Z

   On Sat, Dec 8, 2012 at 10:40 AM, Daniel Farina <daniel@heroku.com> wrote:
   > * A contrib, pg_logcollectdup.  This contrib lets one forward logs to
   >   a named pipe specified in postgresql.conf.
   
   I have revised this part in the attached patch.  It's some error
   handling in a case of user error, and the previous demo script and
   narrative precepts are still the same.
   
   --
   fdr
   

3. Re: Sketch of a Hook into the Logging Collector

   Daniel Farina <daniel@heroku.com> — 2012-12-10T23:15:29Z

   On Sat, Dec 8, 2012 at 10:40 AM, Daniel Farina <daniel@heroku.com> wrote:
   > Hello all,
   >
   > I am approaching this from the angle of increasing power by exposing
   > the log collector ("syslogger") pipe protocol.
   
   I just spotted a better, already-committed patch.  Thanks to Hannu for
   pointing it out:
   
   https://commitfest.postgresql.org/action/patch_view?id=717
   
   I'll retract this patch, unless someone finds it interesting for some reason.
   
   --
   fdr