Thread

more contrib: log rotator

Andrew Sullivan <andrew@libertyrms.info> — 2003-04-03T15:21:52Z

Since now is the time for contrib/ flamewars, this seemed a good time
to suggest this.

My colleague, Sorin Iszlai, wrote us a little program for rotating
our Postgres logs.  It reads stdout and stderr, and sends them to
different files (and rotates them as necessary).  It is currently
hand-configureable (i.e. by altering some variables at the top of the
script), and is more or less designed for use in our own environment.

Tom Lane recently mentioned to me that a common complaint is that
postgres doesn't have its own log rotator.  There are, of course,
plenty of good ones, and syslog itself works pretty well for most
people.  But there are still complaints from time to time about the
lack of a "built in" log rotator.

We'd be happy to release our rotator under the PostgreSQL BSD
license, if it would be of use to people.  I was thinking that
perhaps contrib/ would be a good place for it, since the idea is to
reduce complaints that there's no log rotator "included".  

Is anyone interested in having pglog-rotator?

A
-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110

Re: more contrib: log rotator

Tom Lane <tgl@sss.pgh.pa.us> — 2003-04-03T16:07:44Z

Andrew Sullivan <andrew@libertyrms.info> writes:
> Is anyone interested in having pglog-rotator?

FWIW, I saw an early version of pglog-rotator about a year and a half
ago (while consulting for LibertyRMS), and thought at the time that
it was pretty cool.  So I'm for including it ... maybe even as
mainstream instead of contrib.

			regards, tom lane

Re: more contrib: log rotator

Jim Buttafuoco <jim@contactbda.com> — 2003-04-03T18:05:51Z

Would the plan be to add it to pg_ctl?


> Andrew Sullivan <andrew@libertyrms.info> writes:
> > Is anyone interested in having pglog-rotator?
> 
> FWIW, I saw an early version of pglog-rotator about a year and a half
> ago (while consulting for LibertyRMS), and thought at the time that
> it was pretty cool.  So I'm for including it ... maybe even as
> mainstream instead of contrib.
> 
> 			regards, tom lane
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly

Re: more contrib: log rotator

Tom Lane <tgl@sss.pgh.pa.us> — 2003-04-03T18:41:08Z

"Jim Buttafuoco" <jim@contactbda.com> writes:
> Would the plan be to add it to pg_ctl?

You would not actually have to: you could just pipe pg_ctl's output to
pglog-rotator.  But I think it'd be cool if pg_ctl had an option to use
pglog-rotator, or maybe even adopt it as standard behavior.

I think we would have to make the rotator script be mainstream rather
than contrib if we wanted pg_ctl to use it directly.  That was why I was
thinking maybe mainstream ...

Andrew, could you toss up the script on pgsql-patches just so people can
take a look?  Then we could think more about where to go with it.

			regards, tom lane

Re: more contrib: log rotator

Dave Cramer <dave@fastcrypt.com> — 2003-04-03T19:12:03Z

Does this log rotator do something that apache's doesn't?

Dave
On Thu, 2003-04-03 at 13:41, Tom Lane wrote:
> "Jim Buttafuoco" <jim@contactbda.com> writes:
> > Would the plan be to add it to pg_ctl?
> 
> You would not actually have to: you could just pipe pg_ctl's output to
> pglog-rotator.  But I think it'd be cool if pg_ctl had an option to use
> pglog-rotator, or maybe even adopt it as standard behavior.
> 
> I think we would have to make the rotator script be mainstream rather
> than contrib if we wanted pg_ctl to use it directly.  That was why I was
> thinking maybe mainstream ...
> 
> Andrew, could you toss up the script on pgsql-patches just so people can
> take a look?  Then we could think more about where to go with it.
> 
> 			regards, tom lane
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
-- 
Dave Cramer <dave@fastcrypt.com>
Cramer Consulting

Re: more contrib: log rotator

Andrew Sullivan <andrew@libertyrms.info> — 2003-04-03T21:12:14Z

On Thu, Apr 03, 2003 at 01:41:08PM -0500, Tom Lane wrote:
> You would not actually have to: you could just pipe pg_ctl's output to
> pglog-rotator.  But I think it'd be cool if pg_ctl had an option to use
> pglog-rotator, or maybe even adopt it as standard behavior.

It's currently built to call a program, and read its stdout and
stderr, rather than acting as a pipe.  I guess it shouldn't be too
hard to modify, though.  We actually call the postmaster directly
with it, so we use it as a replacement for pg_ctl at startup.

> Andrew, could you toss up the script on pgsql-patches just so people can
> take a look?  Then we could think more about where to go with it.

Ok, I sent it.

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110

Re: more contrib: log rotator

Andrew Sullivan <andrew@libertyrms.info> — 2003-04-03T21:14:19Z

On Thu, Apr 03, 2003 at 02:12:03PM -0500, Dave Cramer wrote:
> Does this log rotator do something that apache's doesn't?

Probably not.  This was just easier for us.

A little information might be handy here: we run postgres nder a
hosted environment, and we do not have root on the relevant boxes. 
So installing anything even a little complicated means building
everything ourselves.  As a result, we end up re-creating plenty of
functionality just to make it easy to install.

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110

Re: more contrib: log rotator

Andrew Sullivan <andrew@libertyrms.info> — 2003-04-04T03:00:23Z

On Thu, Apr 03, 2003 at 01:41:08PM -0500, Tom Lane wrote:
> Andrew, could you toss up the script on pgsql-patches just so people can
> take a look?  Then we could think more about where to go with it.

Ok, the first try failed (of course) because I wasn't subscribed. 
Should be there now, though.

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110

Re: more contrib: log rotator

Peter Eisentraut <peter_e@gmx.net> — 2003-04-04T15:13:13Z

Andrew Sullivan writes:

> Is anyone interested in having pglog-rotator?

What would get me a whole lot more excited is if the server could write
directly to a file and do its own rotating (or at least reopening of
files).

Considering that your rotator is tailored to a rather specific setup, it
doesn't do anything better compared to established ones, it prevents the
use of pg_ctl, it's written in Perl, and it doesn't do anything for
Windows users, I think it's not suitable for a general audience.

-- 
Peter Eisentraut   peter_e@gmx.net

Re: more contrib: log rotator

scott.marlowe <scott.marlowe@ihs.com> — 2003-04-04T16:16:39Z

On Fri, 4 Apr 2003, Peter Eisentraut wrote:

> Andrew Sullivan writes:
> 
> > Is anyone interested in having pglog-rotator?
> 
> What would get me a whole lot more excited is if the server could write
> directly to a file and do its own rotating (or at least reopening of
> files).
> 
> Considering that your rotator is tailored to a rather specific setup, it
> doesn't do anything better compared to established ones, it prevents the
> use of pg_ctl, it's written in Perl, and it doesn't do anything for
> Windows users, I think it's not suitable for a general audience.

That said, a log rotation capability built right into pg_ctl or 
thereabouts would be a very nice feature.  I.e. 'pg_ctl -r 86400 -l 
$PGDATA/logs/pgsql start'

where -r is the rotation period in seconds.  If it's an external program 
that pg_ctl calls that's fine, and it could even just be a carbon copy of 
apache's log rotater if their license is compatible (isn't it?)

Re: more contrib: log rotator

Andrew Sullivan <andrew@libertyrms.info> — 2003-04-04T16:35:09Z

On Fri, Apr 04, 2003 at 09:16:39AM -0700, scott.marlowe wrote:
> where -r is the rotation period in seconds.  If it's an external program 

Ours rotates based on size rather than time.  I can see some
advantages to the time-based approach, but if you have wide
variations in traffic, you run the risk of rotating over useful files
with more or less empty ones if you use it.

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110

Re: more contrib: log rotator

Andrew Sullivan <andrew@libertyrms.info> — 2003-04-04T16:37:55Z

On Fri, Apr 04, 2003 at 05:13:13PM +0200, Peter Eisentraut wrote:
> use of pg_ctl, it's written in Perl, and it doesn't do anything for
> Windows users, I think it's not suitable for a general audience.

It doesn't prevent the use of pg_ctl, although it does indeed prevent
the use of pg_ctl for startup.  

I'm not sufficiently familiar with Windows to know how this does or
does not help them.  Could you elaborate?  And what's wrong with
Perl?

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110

Re: more contrib: log rotator

Ed L. <pgsql@bluepolka.net> — 2003-04-04T17:04:25Z

On Friday April 4 2003 9:16, scott.marlowe wrote:
>
> That said, a log rotation capability built right into pg_ctl or
> thereabouts would be a very nice feature.  I.e. 'pg_ctl -r 86400 -l
> $PGDATA/logs/pgsql start'
>
> where -r is the rotation period in seconds.  If it's an external program
> that pg_ctl calls that's fine, and it could even just be a carbon copy of
> apache's log rotater if their license is compatible (isn't it?)

By way of feature ideas, one very convenient but not widely used feature of 
Apache's log rotator is the ability to specify a strftime() format string 
for the file extension.  For example, if I want to have my logs rollover 
every 24 hours and be named log.Mon, log.Tue, log.Wed, I say something like

	pg_ctl start | rotatelogs 86400 "%a"

This causes the logs to overwrite themselves every seven days, taking log 
maintenance time to very near zero.  We also customized our use of it to 
allow us to automatically move existing logs out of the way to "log.1", 
"log.2", or to simply overwrite existing logs.

Ed

Re: more contrib: log rotator

Ed L. <pgsql@bluepolka.net> — 2003-04-04T17:06:42Z

On Friday April 4 2003 10:04, Ed L. wrote:
> By way of feature ideas, one very convenient but not widely used feature
> of Apache's log rotator is the ability to specify a strftime() format
> string for the file extension.  For example, if I want to have my logs
> rollover every 24 hours and be named log.Mon, log.Tue, log.Wed, I say
> something like
>
> 	pg_ctl start | rotatelogs 86400 "%a"

More accurately, something like this:

	pg_ctl start | rotatelogs 86400 "log.%a"

Ed

Re: more contrib: log rotator

scott.marlowe <scott.marlowe@ihs.com> — 2003-04-04T17:10:22Z

On Fri, 4 Apr 2003, Andrew Sullivan wrote:

> On Fri, Apr 04, 2003 at 09:16:39AM -0700, scott.marlowe wrote:
> > where -r is the rotation period in seconds.  If it's an external program 
> 
> Ours rotates based on size rather than time.  I can see some
> advantages to the time-based approach, but if you have wide
> variations in traffic, you run the risk of rotating over useful files
> with more or less empty ones if you use it.

I would want time based for sure, and I can see the use for size based 
splitting as well.  I wouldn't be hard to have it do both would it?

I just like the idea of it being one of the dozens or so options for 
pg_ctl so it's painless to use for joe six pack.

pg_ctl -r 86400 -l $PGDATA/logs/pgsql

where -r is the rotation period

OR

pg_ctl -f 10M -l $PGDATA/logs/pgsql

where -f is the max file size of a log

I'd recommend that the nameing convnention should probably 
be:

filenamespec.timestamp, like:  $PGDATA/logs/pgsql.1049414400

for time rotated logs, and 

filename.incnumber like:  $PGDATa/logs/pgsql.0000000001

Re: more contrib: log rotator

Jan Wieck <janwieck@yahoo.com> — 2003-04-04T17:13:37Z

Peter Eisentraut wrote:
> 
> Andrew Sullivan writes:
> 
> > Is anyone interested in having pglog-rotator?
> 
> What would get me a whole lot more excited is if the server could write
> directly to a file and do its own rotating (or at least reopening of
> files).

From a technical point of view I don't think that is desirable. The
entire log traffic would have to be routed through the postmaster, as it
is in LibertyRMS's log rotator now through the perl script. And we
really try to keep everything outside the postmaster that does not
absolutely have to be in there for stability reasons.

We can discuss if the log rotator should be a child process of the
postmaster or the other way round, but that will not change the flow of
bytes between the processes in any way.

I would say it's better the way it is, because it does not pollute the
postmasters wait logic with another exception.

My ideal solution would be to integrate the log rotators functionality
into a C version of pg_ctl that forks and detaches from the control
terminal in the way, daemons should.

Jan

-- 
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #

Re: more contrib: log rotator

Tom Lane <tgl@sss.pgh.pa.us> — 2003-04-04T17:19:49Z

Peter Eisentraut <peter_e@gmx.net> writes:
> What would get me a whole lot more excited is if the server could write
> directly to a file and do its own rotating (or at least reopening of
> files).

AFAICS, the only practical way to do this is to have a single process
collecting the stdout/stderr from the postmaster and all its children.
pglog-rotator is one implementation of that approach.

I too would rather this functionality were integrated into the server,
but I haven't noticed anyone stepping up to the plate to do it.

> Considering that your rotator is tailored to a rather specific setup, it
> doesn't do anything better compared to established ones, it prevents the
> use of pg_ctl, it's written in Perl, and it doesn't do anything for
> Windows users, I think it's not suitable for a general audience.

These might be good arguments for not putting it into the mainstream,
but I don't think they have any force if we consider it for contrib.

I feel we really ought to have *some* rotator included in the standard
distro, just so that the Admin Guide can point to a concrete solution
instead of having to arm-wave about what you can get off the net.
If someone can offer a better alternative than Andrew's, great, let's
see it.

			regards, tom lane

Re: more contrib: log rotator

Ed L. <pgsql@bluepolka.net> — 2003-04-04T18:10:16Z

On Friday April 4 2003 10:19, Tom Lane wrote:
>
> I feel we really ought to have *some* rotator included in the standard
> distro, just so that the Admin Guide can point to a concrete solution
> instead of having to arm-wave about what you can get off the net.
> If someone can offer a better alternative than Andrew's, great, let's
> see it.

Out of curiosity, are there issues preventing inclusion of Apache's log 
rotation code?  It seems you'd be hard-pressed to find a more 
battle-hardened log rotator.

Obviously some people also wish to rotate based on log file size, so adding 
both to contrib at least seems sensible.

Ed

Re: more contrib: log rotator

scott.marlowe <scott.marlowe@ihs.com> — 2003-04-04T18:41:04Z

On Fri, 4 Apr 2003, Ed L. wrote:

> On Friday April 4 2003 10:19, Tom Lane wrote:
> >
> > I feel we really ought to have *some* rotator included in the standard
> > distro, just so that the Admin Guide can point to a concrete solution
> > instead of having to arm-wave about what you can get off the net.
> > If someone can offer a better alternative than Andrew's, great, let's
> > see it.
> 
> Out of curiosity, are there issues preventing inclusion of Apache's log 
> rotation code?  It seems you'd be hard-pressed to find a more 
> battle-hardened log rotator.
> 
> Obviously some people also wish to rotate based on log file size, so adding 
> both to contrib at least seems sensible.

OK, I'm playing with the pg_ctl script that comes with 7.3, and trying to 
make it startup with apaches rotatelog script, but this line won't pipe 
output.  I'm a total noob at bash shell scripting, so please feel free to 
snicker when you answer.

rotatelogs is in my path and all, it just never sees it.

"$po_path" ${1+"$@"} </dev/null | $PGPATH/rotatelogs $logfile $DURATION 2>&1 &

Re: more contrib: log rotator

Tom Lane <tgl@sss.pgh.pa.us> — 2003-04-04T18:58:46Z

"scott.marlowe" <scott.marlowe@ihs.com> writes:
> rotatelogs is in my path and all, it just never sees it.

You mean the command fails?  Or just that it doesn't capture output?

> "$po_path" ${1+"$@"} </dev/null | $PGPATH/rotatelogs $logfile $DURATION 2>&1 &

Most if not all of the postmaster's log output goes to stderr, so you'd need

"$po_path" ${1+"$@"} </dev/null 2>&1 | $PGPATH/rotatelogs ...

to have any hope of useful results.

			regards, tom lane

Re: more contrib: log rotator

Ed L. <pgsql@bluepolka.net> — 2003-04-04T19:10:13Z

On Friday April 4 2003 11:58, Tom Lane wrote:
> "scott.marlowe" <scott.marlowe@ihs.com> writes:
> > rotatelogs is in my path and all, it just never sees it.
>
> You mean the command fails?  Or just that it doesn't capture output?
>
> > "$po_path" ${1+"$@"} </dev/null | $PGPATH/rotatelogs $logfile $DURATION
> > 2>&1 &
>
> Most if not all of the postmaster's log output goes to stderr, so you'd
> need
>
> "$po_path" ${1+"$@"} </dev/null 2>&1 | $PGPATH/rotatelogs ...
>
> to have any hope of useful results.

Hmmm.  I would have agreed 2>&1 was needed, too, but this command seems to 
routinely capture all output, including ERRORs:

	nohup pg_ctl start | nohup rotatelogs server_log.%a 86400

Ed

Re: more contrib: log rotator

scott.marlowe <scott.marlowe@ihs.com> — 2003-04-04T19:49:37Z

On Fri, 4 Apr 2003, Tom Lane wrote:

> "scott.marlowe" <scott.marlowe@ihs.com> writes:
> > rotatelogs is in my path and all, it just never sees it.
> 
> You mean the command fails?  Or just that it doesn't capture output?

The database starts, but rotatelogs doesn't get run.  I.e. it's just like 
everything after the | symbol isn't there.

Re: more contrib: log rotator

Tom Lane <tgl@sss.pgh.pa.us> — 2003-04-04T20:13:44Z

"Ed L." <pgsql@bluepolka.net> writes:
> Hmmm.  I would have agreed 2>&1 was needed, too, but this command seems to 
> routinely capture all output, including ERRORs:
> 	nohup pg_ctl start | nohup rotatelogs server_log.%a 86400

That's 'cause pg_ctl internally redirects the postmaster's stderr.

			regards, tom lane

Re: more contrib: log rotator

scott.marlowe <scott.marlowe@ihs.com> — 2003-04-04T21:17:41Z

On Fri, 4 Apr 2003, Ed L. wrote:

> On Friday April 4 2003 11:58, Tom Lane wrote:
> > "scott.marlowe" <scott.marlowe@ihs.com> writes:
> > > rotatelogs is in my path and all, it just never sees it.
> >
> > You mean the command fails?  Or just that it doesn't capture output?
> >
> > > "$po_path" ${1+"$@"} </dev/null | $PGPATH/rotatelogs $logfile $DURATION
> > > 2>&1 &
> >
> > Most if not all of the postmaster's log output goes to stderr, so you'd
> > need
> >
> > "$po_path" ${1+"$@"} </dev/null 2>&1 | $PGPATH/rotatelogs ...
> >
> > to have any hope of useful results.
> 
> Hmmm.  I would have agreed 2>&1 was needed, too, but this command seems to 
> routinely capture all output, including ERRORs:
> 
> 	nohup pg_ctl start | nohup rotatelogs server_log.%a 86400

OK, So I tried putting the 2>&1 before the | and all.  No matter what I 
try, every from the | on is ignored.  ps doesn't show it, and neither does 
pg_ctl status.  Both show a command line of 
/usr/local/pgsql/bin/postmaster as the only input to start the server.

Now, the thing is, I've tried this with hardcoded values, like:

"$po_path" ${1+"$@"} </dev/null 2>&1 /usr/local/pgsql/bin/rotatelogs 
/mnt/d1/data/logs/pglog 86400

where I know the logs directory exists.  It works if I do:

pg_ctl start | rotatelogs $PGDATA/pglog 86400 2>1&

and puts the log files there.

I've copied rotatelogs into the /usr/local/pgsql/bin directory as well.

So, I'm thinking this is my weakness in shell scripting that's getting me 
here, and that the shell is eating the |, not passing it out with the 
postmaster to be used when it starts.

Re: more contrib: log rotator

Ed L. <pgsql@bluepolka.net> — 2003-04-04T21:45:07Z

On Friday April 4 2003 2:17, scott.marlowe wrote:
>
> OK, So I tried putting the 2>&1 before the | and all.  No matter what I
> try, every from the | on is ignored.  ps doesn't show it, and neither
> does pg_ctl status.  Both show a command line of
> /usr/local/pgsql/bin/postmaster as the only input to start the server.

Not clear if you're looking at it this way or if this is your problem, but 
you can't really tell there is log rotation going on just by grepping ps 
for postmaster because ps does not typically show the postmaster and the 
rotatelogs together on the same line.  I wouldn't expect pg_ctl status to 
know anything at all about rotatelogs when you pipe it like this.

Ed

Re: more contrib: log rotator

scott.marlowe <scott.marlowe@ihs.com> — 2003-04-04T22:34:22Z

On Fri, 4 Apr 2003, Ed L. wrote:

> On Friday April 4 2003 2:17, scott.marlowe wrote:
> >
> > OK, So I tried putting the 2>&1 before the | and all.  No matter what I
> > try, every from the | on is ignored.  ps doesn't show it, and neither
> > does pg_ctl status.  Both show a command line of
> > /usr/local/pgsql/bin/postmaster as the only input to start the server.
> 
> Not clear if you're looking at it this way or if this is your problem, but 
> you can't really tell there is log rotation going on just by grepping ps 
> for postmaster because ps does not typically show the postmaster and the 
> rotatelogs together on the same line.  I wouldn't expect pg_ctl status to 
> know anything at all about rotatelogs when you pipe it like this.

Hey, do you guys think that a setting of silent_mode = false might affect 
no log files getting created?

I had it right as soon as I added Tom's recommended 2>&1 but spent another 
30 minutes figuring out why my log file wasn't getting created / filled.   

Thanks for the help.

Re: more contrib: log rotator

Tom Lane <tgl@sss.pgh.pa.us> — 2003-04-04T22:42:16Z

"scott.marlowe" <scott.marlowe@ihs.com> writes:
> Hey, do you guys think that a setting of silent_mode = false might affect 
> no log files getting created?

No, but setting it to true would be bad news.

			regards, tom lane

Re: more contrib: log rotator

scott.marlowe <scott.marlowe@ihs.com> — 2003-04-04T22:50:57Z

On Fri, 4 Apr 2003, Tom Lane wrote:

> "scott.marlowe" <scott.marlowe@ihs.com> writes:
> > Hey, do you guys think that a setting of silent_mode = false might affect 
> > no log files getting created?
> 
> No, but setting it to true would be bad news.

That's what I'd meant actually.  I had to turn of silent mode...  You know 
you're having a bad day when your email explaining how stupid you are is 
factually incorrect. :-)

 If anyone wants the diff, here it is:

22c22
<   $CMDNAME start   [-w] [-D DATADIR] [-s] [-l FILENAME] [-o \"OPTIONS\"]
---
>   $CMDNAME start   [-w] [-D DATADIR] [-s] [-r DURATION] [-l FILENAME] 
[-o \"OPTIONS\"]
39a40,41
>   -r DURATION             invoke log rotation with DURATION seconds
>                           between rotation of files.
155a158,161
>       -r)
>           DURATION="$2"
>           shift
>           ;;
336c342,346
<         "$po_path" ${1+"$@"} </dev/null >>$logfile 2>&1 &
---
>         if [ -n "$DURATION" ]; then
>             "$po_path" ${1+"$@"} </dev/null 2>&1| $PGPATH/rotatelogs 
$logfile $DURATION 2>&1 &
>         else
>             "$po_path" ${1+"$@"} </dev/null >>$logfile 2>&1 &
>         fi

Re: more contrib: log rotator

Peter Eisentraut <peter_e@gmx.net> — 2003-04-05T00:03:03Z

Tom Lane writes:

> AFAICS, the only practical way to do this is to have a single process
> collecting the stdout/stderr from the postmaster and all its children.

I think not.  It's a little tricky handling it directly in the child
processes, but it's been done before.

> If someone can offer a better alternative than Andrew's, great, let's
> see it.

How about the attached one, which I floated a while ago but which didn't
generate much interest.

-- 
Peter Eisentraut   peter_e@gmx.net

Re: more contrib: log rotator

Tom Lane <tgl@sss.pgh.pa.us> — 2003-04-05T00:22:58Z

Peter Eisentraut <peter_e@gmx.net> writes:
> Tom Lane writes:
>> AFAICS, the only practical way to do this is to have a single process
>> collecting the stdout/stderr from the postmaster and all its children.

> I think not.  It's a little tricky handling it directly in the child
> processes, but it's been done before.

A "little" tricky?  Thanks, but no thanks ... for one thing, there'd be
no easy way to know when all the children had switched over to writing
the new file.  Also, at least for not-too-long messages, writing on a
single pipe gives atomicity guarantees that AFAIK do not exist when
writing a file through multiple independently opened descriptors.  In
the latter case I think we'd have lots of trouble with interleaving of
messages from different backends.

>> If someone can offer a better alternative than Andrew's, great, let's
>> see it.

> How about the attached one, which I floated a while ago but which didn't
> generate much interest.

Seems like a good bare-bones file writer; but how about all those
frammishes that people ask for like generating date-based filenames,
switching every so many bytes, etc?  Also, it'd be nice not to be
dependent on a cron job to tickle the switchover.

I do think there's an efficiency argument for having the log writer
coded in C, so starting with what you have here and building up might
be a better idea than starting with Andrew's perl script.  But the
important thing in my mind is to get something in there.

We should also take a look at Apache's rotator to see if there's any need
to reinvent the wheel at all.  I have not seen it, am not even sure what
it's written in...

			regards, tom lane

Re: more contrib: log rotator

Jon Jensen <jon@endpoint.com> — 2003-04-05T14:23:25Z

On Fri, 4 Apr 2003, Tom Lane wrote:

> We should also take a look at Apache's rotator to see if there's any need
> to reinvent the wheel at all.  I have not seen it, am not even sure what
> it's written in...

It's written in 140 lines of C (blank lines and all), and has been very
solid in my experience. I don't know of any deficiencies that would
warrant rewriting it.

Jon

Re: more contrib: log rotator

Peter Eisentraut <peter_e@gmx.net> — 2003-04-06T21:03:27Z

Tom Lane writes:

> Seems like a good bare-bones file writer; but how about all those
> frammishes that people ask for like generating date-based filenames,
> switching every so many bytes, etc?  Also, it'd be nice not to be
> dependent on a cron job to tickle the switchover.

Linux systems have a standard system log rotation mechanism (see
logrotate(8)), which can rotate logs by size and time and has a number of
other features.  I would rather depend on that kind of preferred system
mechanism than rolling out our own.  And we already depend on cron for
vacuuming anyway.

-- 
Peter Eisentraut   peter_e@gmx.net

Re: more contrib: log rotator

Andrew Sullivan <andrew@libertyrms.info> — 2003-04-06T22:17:15Z

On Sun, Apr 06, 2003 at 11:03:27PM +0200, Peter Eisentraut wrote:
> Linux systems have a standard system log rotation mechanism (see
> logrotate(8)), which can rotate logs by size and time and has a number of
> other features.  I would rather depend on that kind of preferred system

But it's not available on every platform.  And according to my
/usr/share/doc/logrotate/copyright, it's GPL, so you can't
redistribute it with PostgreSQL.  Building it in was one of the
criteria, I thought

> And we already depend on cron for vacuuming anyway.

But that dependency is actually a liability, because there's no way
to say, "Hey, I really need to be vacuumed now." We (at Liberty) are
currently going through a remarkable number of hoops trying to get
around that very limitation

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110

Re: more contrib: log rotator

Peter Eisentraut <peter_e@gmx.net> — 2003-04-06T22:42:34Z

Andrew Sullivan writes:

> But it's not available on every platform.  And according to my
> /usr/share/doc/logrotate/copyright, it's GPL, so you can't
> redistribute it with PostgreSQL.  Building it in was one of the
> criteria, I thought

My point was that log file rotation should be left up to the system
administrator.  Look at other servers on your system (SMTP, DNS,
whatever).  How do they handle it?

> > And we already depend on cron for vacuuming anyway.
>
> But that dependency is actually a liability, because there's no way
> to say, "Hey, I really need to be vacuumed now."

psql -c 'VACUUM' ?

-- 
Peter Eisentraut   peter_e@gmx.net

Re: more contrib: log rotator

Andrew Sullivan <andrew@libertyrms.info> — 2003-04-06T22:54:52Z

On Mon, Apr 07, 2003 at 12:42:34AM +0200, Peter Eisentraut wrote:
> 
> My point was that log file rotation should be left up to the system
> administrator.  Look at other servers on your system (SMTP, DNS,
> whatever).  How do they handle it?

PostgreSQL is not a system process, and I think it's a mistake to
assume that it is.  We, for instance, do not have root on the
machines we use.  It's important to assume that users needn't be
system administrators to use the system.

I suppose, however, you could make the argument that log rotation
should be the responisibility of the adminisistrator of the
PostgreSQL server.  But that just amounts to an argument that nothing
needs to be done: as we see, there are lots of log management
facilities on offer, and none of them are included with PostgreSQL. 
I don't feel strongly about which one to use.  But since people
frequently complain that the feature is not available in PostgreSQL,
it does seem that, if it's not too much trouble, adding the feature
is worth it.

> > But that dependency is actually a liability, because there's no way
> > to say, "Hey, I really need to be vacuumed now."
> 
> psql -c 'VACUUM' ?

I meant on the part of the back end.  If you have a busy system on
which some tables need very frequent vacuuming, but it gets
unpredictable traffi, you don't just want to say, "Heck, let's vacuum
every hour."  You want to know _actually_ whether the table needs
vacuuming.

So you come up with a bunch of profiles, &c., and do a pile of work
to figure out which tables really need attention.  But that's not
free: it takes cycles on the machine, and the hypotheical case is one
in which the machine is already under heavy load.  So cron is not
really the answer.

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110

Re: more contrib: log rotator

scott.marlowe <scott.marlowe@ihs.com> — 2003-04-07T15:35:07Z

On Sun, 6 Apr 2003, Peter Eisentraut wrote:

> Tom Lane writes:
> 
> > Seems like a good bare-bones file writer; but how about all those
> > frammishes that people ask for like generating date-based filenames,
> > switching every so many bytes, etc?  Also, it'd be nice not to be
> > dependent on a cron job to tickle the switchover.
> 
> Linux systems have a standard system log rotation mechanism (see
> logrotate(8)), which can rotate logs by size and time and has a number of
> other features.  I would rather depend on that kind of preferred system
> mechanism than rolling out our own.  And we already depend on cron for
> vacuuming anyway.

How about we set up configure to check what we're on and what's available, 
(i.e. rotatelogs, logrotate, joesbiglogrotatorscript, etc...) and 
configure pg_ctl to use one of them?  It's a good probability that most 
flavors of Unix have log rotators of some kind as a built in, and we can 
include a standard one as well.

That way, if you want to use the same log rotator with postgresql as you 
use with the rest of your system, you can, and if you just want the built 
in one, you can use it, and if you don't want any log rotation, everything 
still works the same as before.

Re: more contrib: log rotator

Lamar Owen <lamar.owen@wgcr.org> — 2003-04-07T15:45:07Z

On Sunday 06 April 2003 18:54, Andrew Sullivan wrote:
> On Mon, Apr 07, 2003 at 12:42:34AM +0200, Peter Eisentraut wrote:
> > My point was that log file rotation should be left up to the system
> > administrator.  Look at other servers on your system (SMTP, DNS,
> > whatever).  How do they handle it?

> PostgreSQL is not a system process, and I think it's a mistake to
> assume that it is.  We, for instance, do not have root on the
> machines we use.  It's important to assume that users needn't be
> system administrators to use the system.

I personally believe that making the assumption that PostgreSQL is not a 
system process is wrong.  One can run system services as a normal user (in 
fact, it is recommended that as few system services as is possible should run 
as root); but the fact that a daemon is running as a normal user doesn't make 
it not a system process. But that's just a difference of system 
administration opinion.

However, I can see the utility of a bundled simple log rotator.  The key word 
is simple -- we have the full-fledged route now, called syslog.  And if 
someone needs a better logrotator they can certainly get one of the many that 
are already available.

At the same time I don't necessarily want such a log rotator to be the 
default.  We have syslog as the default.  If someone has the particular need 
for a stderr/stdout log rotator, then let it be a configure option.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Re: more contrib: log rotator

Peter Eisentraut <peter_e@gmx.net> — 2003-04-07T17:06:07Z

Andrew Sullivan writes:

> PostgreSQL is not a system process, and I think it's a mistake to
> assume that it is.

The point is that PostgreSQL should fit nicely with the customs of the
system that it runs on.  This starts with the oft-discussed file system
layout, the use of syslog in the first place, using 'cron' and 'at'
instead of rolling our own mechanisms to schedule jobs, as is occasionally
requested, fitting in with the startup scripts system, and so on.

> I suppose, however, you could make the argument that log rotation
> should be the responisibility of the adminisistrator of the
> PostgreSQL server.  But that just amounts to an argument that nothing
> needs to be done: as we see, there are lots of log management
> facilities on offer, and none of them are included with PostgreSQL.

That is not the argument.  What we need to do is to make it *possible* to
rotate the logs without shutting down the server, not (necessarily) do the
rotation ourselves.  How can we even begin to do that?  Do we need to
invent a configuration language that can control when to rotate, where to
move the old logs, when to delete the even older logs, etc.?

> I meant on the part of the back end.  If you have a busy system on
> which some tables need very frequent vacuuming, but it gets
> unpredictable traffi, you don't just want to say, "Heck, let's vacuum
> every hour."  You want to know _actually_ whether the table needs
> vacuuming.

That is an argument that manual vacuum is a liability, not the use of
cron for it.

-- 
Peter Eisentraut   peter_e@gmx.net

Re: more contrib: log rotator

Your Name <cbbrowne@cbbrowne.com> — 2003-04-07T17:40:41Z

Lamar Owen wrote:
> On Sunday 06 April 2003 18:54, Andrew Sullivan wrote:
> > On Mon, Apr 07, 2003 at 12:42:34AM +0200, Peter Eisentraut wrote:
> > > My point was that log file rotation should be left up to the system
> > > administrator.  Look at other servers on your system (SMTP, DNS,
> > > whatever).  How do they handle it?
> 
> > PostgreSQL is not a system process, and I think it's a mistake to
> > assume that it is.  We, for instance, do not have root on the
> > machines we use.  It's important to assume that users needn't be
> > system administrators to use the system.

> I personally believe that making the assumption that PostgreSQL is not
> a system process is wrong.  One can run system services as a normal
> user (in fact, it is recommended that as few system services as is
> possible should run as root); but the fact that a daemon is running as
> a normal user doesn't make it not a system process. But that's just a
> difference of system administration opinion.

I think the mistake lies in making the "design" assumption that
PostgreSQL is either one or the other.

- There are contexts in which it forcibly is "systemy," such as when it
  is used for password authentication using something like PAM.  In that
  case, whatever userid it runs as, it's a forcible "system" dependancy.
  Users can't log in until PostgreSQL is running.

- There are contexts where it will run as a "part of the system," as is
  typically the case when someone uses "apt-get install postgresql" or
  "rpm -i postgres*.rpm"

- In a "hosted" environment, it may be unacceptable to, in any manner,
  treat PostgreSQL or any related services as "part of the system."
  cron obviously *is* a "part of the system," but if you're not the
  system administrator, you may have /no/ ability to connect in to
  "system" logging services.  (In the environment where 
  pgrotatelog runs, that is indeed the case.)

These are /all/ legitimate scenarios for PostgreSQL to be in use.

--> Assuming PostgreSQL /is/ a system process is wrong.
--> Assuming PostgreSQL /is not/ a system process is wrong.

There are situations where either can be true, and it is vital for
PostgreSQL to be able to support both.
--
(concatenate 'string "cbbrowne" "@ntlug.org")
http://www.ntlug.org/~cbbrowne/unix.html
"There  I  was,  lying,  cheating  and back-stabbing  my  way  up  the
corporate ladder, feeling pretty  darn good about myself, when someone
told me  the 'J' in 'WWJD'  meant *Jesus* I thought  it meant *Judas*!
Hoo boy, am I red in the face!"

Re: more contrib: log rotator

ngpg@grymmjack.com — 2003-05-06T02:40:51Z

multilog


andrew@libertyrms.info (Andrew Sullivan) wrote:
> Since now is the time for contrib/ flamewars, this seemed a good time
> to suggest this.
> 
...
> 
> Is anyone interested in having pglog-rotator?

Re: more contrib: log rotator

Your Name <cbbrowne@cbbrowne.com> — 2003-05-24T02:42:58Z

Peter Eisentraut writes:
> I think not.  It's a little tricky handling it directly in the child
> processes, but it's been done before.

Certainly has...

In the cfengine2 code base, the relevant file is "rotate.c"; it
essentially attaches the file descriptor to a new file "on the fly," and
does so as a separate process.  The code apparently works on NT, too;
there is a comment that indicates that they use chown() rather than
fchown() because the latter doesn't exist on NT.

In fact, that file is well worth taking a look at for strategies on
this.  It's GPLed code, and so may not be suitable for integration, but
there are doubtless some useful techniques to be seen...
--
wm(X,Y):-write(X),write('@'),write(Y). wm('cbbrowne','cbbrowne.com').
http://cbbrowne.com/info/languages.html
"Microsoft is sort of a mixture between the Borg and the
Ferengi. Combine the Borg marketing with Ferengi networking..."
-- Andre Beck in dcouln