Thread

1. Log rotation?

   Peter Eisentraut <peter_e@gmx.net> — 2001-09-05T01:12:28Z

   With all the great work put into allowing true 24/7 operation, as
   distributed we're still unable to rotate the log file.  While the log file
   tends to be smaller than the database system as a whole, this is still
   going to annoy people because they can't control disk usage without taking
   the server down.
   
   I've been playing with a little program I wrote whose sole purpose is to
   write its stdin to a file and close and reopen that file when it receives
   a signal.  I figured this could work well when integrated transparently
   into pg_ctl.
   
   So, is log rotation a concern?  Is this a reasonable solution?  Other
   ideas?
   
   (No Grand Unified Logging Solutions please.  And no, "use syslog" doesn't
   count.)
   
   -- 
   Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter
   
   
   

2. Re: Log rotation?

   Christopher Kings-Lynne <chriskl@familyhealth.com.au> — 2001-09-06T02:39:56Z

   > So, is log rotation a concern?  Is this a reasonable solution?  Other
   > ideas?
   >
   > (No Grand Unified Logging Solutions please.  And no, "use syslog" doesn't
   > count.)
   
   What's the problem with using newsyslog or logrotate at the moment?  (ie.
   use the system log rotator)
   
   Chris
   
   
   

3. Re: Log rotation?

   Tom Lane <tgl@sss.pgh.pa.us> — 2001-09-06T03:19:45Z

   Peter Eisentraut <peter_e@gmx.net> writes:
   > With all the great work put into allowing true 24/7 operation, as
   > distributed we're still unable to rotate the log file.  While the log file
   > tends to be smaller than the database system as a whole, this is still
   > going to annoy people because they can't control disk usage without taking
   > the server down.
   
   > I've been playing with a little program I wrote whose sole purpose is to
   > write its stdin to a file and close and reopen that file when it receives
   > a signal.  I figured this could work well when integrated transparently
   > into pg_ctl.
   
   Aren't there log-rotation utilities out there already?  (I seem to
   recall mention that Apache has one, for instance.)  Seems like this
   is a wheel we shouldn't have to reinvent.
   
   Also, I kinda thought the long-range solution was to encourage everyone
   to migrate to syslog logging ...
   
   > And no, "use syslog" doesn't count.
   
   Why not?
   
   			regards, tom lane
   
   

4. Re: Log rotation?

   Ian Lance Taylor <ian@airs.com> — 2001-09-06T03:54:44Z

   Tom Lane <tgl@sss.pgh.pa.us> writes:
   
   > > And no, "use syslog" doesn't count.
   > 
   > Why not?
   
   The standard implementations of syslog lose log entries under heavy
   load, because they rely on a daemon which reads from a named pipe with
   a limited buffer space.  This is not acceptable in a production
   system, since heavy load is often just the time you need to see the
   log entries.
   
   It would be possible to implement the syslog(3) interface in a
   different way, of course, which did not use syslogd.  I don't know of
   any such implementation.
   
   (My personal preference these days is an approach like DJB's
   daemontools, which separates the handling of log entries from the
   program doing the logging.)
   
   Ian
   
   

5. Re: Log rotation?

   Tom Lane <tgl@sss.pgh.pa.us> — 2001-09-06T04:04:14Z

   Ian Lance Taylor <ian@airs.com> writes:
   > Tom Lane <tgl@sss.pgh.pa.us> writes:
   >>> And no, "use syslog" doesn't count.
   >> 
   >> Why not?
   
   > The standard implementations of syslog lose log entries under heavy
   > load,
   
   Okay, that's a sufficient answer for that point.
   
   > (My personal preference these days is an approach like DJB's
   > daemontools, which separates the handling of log entries from the
   > program doing the logging.)
   
   That still leads back to my first question, which is whether we can't
   rely on someone else's logrotation code.
   
   			regards, tom lane
   
   

6. Re: Log rotation?

   Peter Eisentraut <peter_e@gmx.net> — 2001-09-06T10:02:55Z

   Tom Lane writes:
   
   > Aren't there log-rotation utilities out there already?  (I seem to
   > recall mention that Apache has one, for instance.)  Seems like this
   > is a wheel we shouldn't have to reinvent.
   
   I'm aware of the Apache rotatelogs utility, but I'm not completely
   satisfied with it.
   
   1.  It tries to do the rotating itself.  I'd rather rely on the OS'
   rotating and archiving facilities.
   
   2.  Only offers a time-based rotate, no manual intervention possible (via
   signal).
   
   3.  We don't want to have to tell people to install Apache and patch their
   pg_ctl.
   
   4.  We don't want to include it in our distribution because the license
   contains an advertisement clause.
   
   It's not like what I wrote is going to look wildly different than theirs.
   There's only so much variation you can put into 100 lines of code.
   
   > > And no, "use syslog" doesn't count.
   >
   > Why not?
   
   1. Might not be available (?)
   
   2. Might not be reliable
   
   3. Might not have root access
   
   4. Not all messages will go through elog.  This is a bug, but not trivial
   to fix.
   
   -- 
   Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter
   
   
   

7. Re: Log rotation?

   Peter Eisentraut <peter_e@gmx.net> — 2001-09-06T10:04:03Z

   Christopher Kings-Lynne writes:
   
   > What's the problem with using newsyslog or logrotate at the moment?  (ie.
   > use the system log rotator)
   
   The postmaster will never close the output file, so you can rotate all you
   want, the original file will never be abandoned.
   
   -- 
   Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter
   
   
   

8. Re: Log rotation?

   D'Arcy Cain <darcy@druid.net> — 2001-09-06T12:33:22Z

   Thus spake Tom Lane
   > Also, I kinda thought the long-range solution was to encourage everyone
   > to migrate to syslog logging ...
   > 
   > > And no, "use syslog" doesn't count.
   > 
   > Why not?
   
   Well, one "why not" might be that syslog is not a guaranteed delivery
   logging system.  It might be good enough for some applications but I
   don't think that it should be forced on everyone.
   
   -- 
   D'Arcy J.M. Cain <darcy@{druid|vex}.net>   |  Democracy is three wolves
   http://www.druid.net/darcy/                |  and a sheep voting on
   +1 416 425 1212     (DoD#0082)    (eNTP)   |  what's for dinner.
   
   

9. Re: Log rotation?

   Tom Lane <tgl@sss.pgh.pa.us> — 2001-09-06T13:51:54Z

   Peter Eisentraut <peter_e@gmx.net> writes:
   > Tom Lane writes:
   >> Aren't there log-rotation utilities out there already?  (I seem to
   >> recall mention that Apache has one, for instance.)  Seems like this
   >> is a wheel we shouldn't have to reinvent.
   
   > I'm aware of the Apache rotatelogs utility, but I'm not completely
   > satisfied with it.
   
   Okay, those are reasonable points.  Given that it's only ~100 lines of
   code, I'll withdraw my objection to rolling our own.  Let's just do it.
   
   			regards, tom lane
   
   

10. Re: Log rotation?

    Matthew Hagerty <mhagerty@voyager.net> — 2001-09-06T14:08:21Z

    At 08:54 PM 9/5/2001 -0700, Ian Lance Taylor wrote:
    >Tom Lane <tgl@sss.pgh.pa.us> writes:
    >
    > > > And no, "use syslog" doesn't count.
    > >
    > > Why not?
    >
    >The standard implementations of syslog lose log entries under heavy
    >load, because they rely on a daemon which reads from a named pipe with
    >a limited buffer space.  This is not acceptable in a production
    >system, since heavy load is often just the time you need to see the
    >log entries.
    >
    >It would be possible to implement the syslog(3) interface in a
    >different way, of course, which did not use syslogd.  I don't know of
    >any such implementation.
    >
    >(My personal preference these days is an approach like DJB's
    >daemontools, which separates the handling of log entries from the
    >program doing the logging.)
    >
    >Ian
    
    Greetings,
    
    Kind of ironic, I have been working on a similar logging system for Apache 
    that works with PostgreSQL, and I just released 2.0-beta last night.  My 
    post to announcements was delayed, but you can check it out here: 
    http://www.digitalstratum.com/pglogd/
    
    If pgLOGd looks like something similar to what you are looking for, I could 
    probably modify it to log for PostgreSQL.  Two of its requirements during 
    development were fast and robust, and similar to what you described above 
    it does not "process" the entries, that is done later.  You also got me 
    thinking that maybe syslogd needs an overhaul too...
    
    Matthew
    
    
    

11. Re: Log rotation?

    Ken Hirsch <kenhirsch@myself.com> — 2001-09-06T14:48:52Z

    You may be interested in
    http://www.ietf.org/internet-drafts/draft-ietf-syslog-reliable-12.txt which
    builds a reliable syslog protocol on top of BEEP.  There are free
    implementations of BEEP in C and Java at http://beepcore.org
    
    ----- Original Message -----
    From: "Matthew Hagerty" <mhagerty@voyager.net>
    To: "Ian Lance Taylor" <ian@airs.com>; "Tom Lane" <tgl@sss.pgh.pa.us>
    Cc: "Peter Eisentraut" <peter_e@gmx.net>; "PostgreSQL Development"
    <pgsql-hackers@postgresql.org>
    Sent: Thursday, September 06, 2001 10:08 AM
    Subject: Re: [HACKERS] Log rotation?
    
    
    > At 08:54 PM 9/5/2001 -0700, Ian Lance Taylor wrote:
    > >Tom Lane <tgl@sss.pgh.pa.us> writes:
    > >
    > > > > And no, "use syslog" doesn't count.
    > > >
    > > > Why not?
    > >
    > >The standard implementations of syslog lose log entries under heavy
    > >load, because they rely on a daemon which reads from a named pipe with
    > >a limited buffer space.  This is not acceptable in a production
    > >system, since heavy load is often just the time you need to see the
    > >log entries.
    > >
    > >It would be possible to implement the syslog(3) interface in a
    > >different way, of course, which did not use syslogd.  I don't know of
    > >any such implementation.
    > >
    > >(My personal preference these days is an approach like DJB's
    > >daemontools, which separates the handling of log entries from the
    > >program doing the logging.)
    > >
    > >Ian
    >
    > Greetings,
    >
    > Kind of ironic, I have been working on a similar logging system for Apache
    > that works with PostgreSQL, and I just released 2.0-beta last night.  My
    > post to announcements was delayed, but you can check it out here:
    > http://www.digitalstratum.com/pglogd/
    >
    > If pgLOGd looks like something similar to what you are looking for, I
    could
    > probably modify it to log for PostgreSQL.  Two of its requirements during
    > development were fast and robust, and similar to what you described above
    > it does not "process" the entries, that is done later.  You also got me
    > thinking that maybe syslogd needs an overhaul too...
    >
    > Matthew
    >
    >
    > ---------------------------(end of broadcast)---------------------------
    > TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
    
    
    

12. Re: Log rotation?

    Christopher Kings-Lynne <chriskl@familyhealth.com.au> — 2001-09-07T02:06:19Z

    Yeah, I use FreeBSD's wonderful newsyslog utility, and I do my logging like
    this:
    
    su -l pgsql -c '[ -d ${PGDATA} ] && exec /usr/local/bin/pg_ctl
    start -s -w -o "-i" -l /var/log/pgsql.log'
    
    And my /etc/newsyslog.conf entry:
    
    /var/log/pgsql.log      pgsql:pgsql     600  3     4096 *     Z
    
    Chris
    
    > -----Original Message-----
    > From: pgsql-hackers-owner@postgresql.org
    > [mailto:pgsql-hackers-owner@postgresql.org]On Behalf Of Peter Eisentraut
    > Sent: Thursday, 6 September 2001 6:04 PM
    > To: Christopher Kings-Lynne
    > Cc: PostgreSQL Development
    > Subject: Re: [HACKERS] Log rotation?
    >
    >
    > Christopher Kings-Lynne writes:
    >
    > > What's the problem with using newsyslog or logrotate at the
    > moment?  (ie.
    > > use the system log rotator)
    >
    > The postmaster will never close the output file, so you can rotate all you
    > want, the original file will never be abandoned.
    >
    > --
    > Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter
    >
    >
    > ---------------------------(end of broadcast)---------------------------
    > TIP 6: Have you searched our list archives?
    >
    > http://www.postgresql.org/search.mpl
    >
    
    
    

13. Re: Log rotation?

    Peter Eisentraut <peter_e@gmx.net> — 2001-09-07T14:04:24Z

    Christopher Kings-Lynne writes:
    
    > Yeah, I use FreeBSD's wonderful newsyslog utility, and I do my logging like
    > this:
    >
    > su -l pgsql -c '[ -d ${PGDATA} ] && exec /usr/local/bin/pg_ctl
    > start -s -w -o "-i" -l /var/log/pgsql.log'
    >
    > And my /etc/newsyslog.conf entry:
    >
    > /var/log/pgsql.log      pgsql:pgsql     600  3     4096 *     Z
    
    Sorry, this does not convey any information to me.  Does this newsyslog
    thing do anything that's so smart that we should know about it?
    
    -- 
    Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter