Thread

Units in postgresql.conf

Peter Eisentraut <peter_e@gmx.net> — 2006-07-20T11:49:36Z

One frequent source of confusion are the different units that the parameters 
in postgresql.conf use.  shared_buffers is in 8 kB, work_mem is in 1 kB; 
bgwriter_delay is in milliseconds, checkpoint_warning is in seconds.  
Obviously, we can't change that without inconveniencing a lot of users.

I think it would be useful to allow units to be added to these settings, for 
example

shared_buffers = 1000kB
checkpoint_warning = 30s

This would also allow

shared_buffers = 512MB

which is a bit cumbersome to calculate right now (you'd need = 65536).

I haven't thought yet how to parse or implement this, but would people find 
this useful?

-- 
Peter Eisentraut
http://developer.postgresql.org/~petere/

Re: Units in postgresql.conf

Jonah H. Harris <jonah.harris@gmail.com> — 2006-07-20T12:04:54Z

On 7/20/06, Peter Eisentraut <peter_e@gmx.net> wrote:
> I think it would be useful to allow units to be added to these settings, for
> example
>
> shared_buffers = 1000kB
> checkpoint_warning = 30s
>
> This would also allow
>
> shared_buffers = 512MB
>
> which is a bit cumbersome to calculate right now (you'd need = 65536).
>
> I haven't thought yet how to parse or implement this, but would people find
> this useful?

I agree, a lot of newbies have issues with the configuration file.  I
have a tiny bit of code (about 20 lines I think) that will handle K,
M, and G suffixes for memory.  It would be equally easy to add S for
seconds, ....

In my code, if no suffix existed, I'd just revert to the default
behavior.  This is probably what we'd want to do in PostgreSQL as
well.

The only issue in PostgreSQL is knowing what the unit conversion and
scaling factor is for each parameter (8K, 1K, milliseconds, etc);
though, this wouldn't be hard to add at all.

-- 
Jonah H. Harris, Software Architect | phone: 732.331.1300
EnterpriseDB Corporation            | fax: 732.331.1301
33 Wood Ave S, 2nd Floor            | jharris@enterprisedb.com
Iselin, New Jersey 08830            | http://www.enterprisedb.com/

Re: Units in postgresql.conf

Gavin Sherry <swm@linuxworld.com.au> — 2006-07-20T12:10:57Z

On Thu, 20 Jul 2006, Peter Eisentraut wrote:

> One frequent source of confusion are the different units that the parameters
> in postgresql.conf use.  shared_buffers is in 8 kB, work_mem is in 1 kB;
> bgwriter_delay is in milliseconds, checkpoint_warning is in seconds.
> Obviously, we can't change that without inconveniencing a lot of users.
>
> I think it would be useful to allow units to be added to these settings, for
> example
>
> shared_buffers = 1000kB
> checkpoint_warning = 30s
>
> This would also allow
>
> shared_buffers = 512MB
>
> which is a bit cumbersome to calculate right now (you'd need = 65536).
>
> I haven't thought yet how to parse or implement this, but would people find
> this useful?

Please! Yes!

Gavin

Re: Units in postgresql.conf

Marko Kreen <markokr@gmail.com> — 2006-07-20T12:13:50Z

On 7/20/06, Peter Eisentraut <peter_e@gmx.net> wrote:
> One frequent source of confusion are the different units that the parameters
> in postgresql.conf use.  shared_buffers is in 8 kB, work_mem is in 1 kB;
> bgwriter_delay is in milliseconds, checkpoint_warning is in seconds.
> Obviously, we can't change that without inconveniencing a lot of users.
>
> I think it would be useful to allow units to be added to these settings, for
> example
>
> shared_buffers = 1000kB
> checkpoint_warning = 30s
>
> This would also allow
>
> shared_buffers = 512MB
>
> which is a bit cumbersome to calculate right now (you'd need = 65536).
>
> I haven't thought yet how to parse or implement this, but would people find
> this useful?

+1.  In addition, that would make conffile self-documenting.

-- 
marko

Re: Units in postgresql.conf

David Fetter <david@fetter.org> — 2006-07-20T14:44:08Z

On Thu, Jul 20, 2006 at 01:49:36PM +0200, Peter Eisentraut wrote:
> One frequent source of confusion are the different units that the parameters 
> in postgresql.conf use.  shared_buffers is in 8 kB, work_mem is in 1 kB; 
> bgwriter_delay is in milliseconds, checkpoint_warning is in seconds.  
> Obviously, we can't change that without inconveniencing a lot of users.
> 
> I think it would be useful to allow units to be added to these settings, for 
> example
> 
> shared_buffers = 1000kB
> checkpoint_warning = 30s
> 
> This would also allow
> 
> shared_buffers = 512MB
> 
> which is a bit cumbersome to calculate right now (you'd need = 65536).
> 
> I haven't thought yet how to parse or implement this, but would
> people find this useful?

Absolutely! :)

Cheers,
D
-- 
David Fetter <david@fetter.org> http://fetter.org/
phone: +1 415 235 3778        AIM: dfetter666
                              Skype: davidfetter

Remember to vote!

Re: Units in postgresql.conf

Zdenek Kotala <zdenek.kotala@sun.com> — 2006-07-20T15:51:27Z

Peter Eisentraut wrote:
> One frequent source of confusion are the different units that the parameters 
> in postgresql.conf use.  shared_buffers is in 8 kB, work_mem is in 1 kB; 
> bgwriter_delay is in milliseconds, checkpoint_warning is in seconds.  
> Obviously, we can't change that without inconveniencing a lot of users.
> 
> I think it would be useful to allow units to be added to these settings, for 
> example
> 
> shared_buffers = 1000kB
> checkpoint_warning = 30s
> 
> This would also allow
> 
> shared_buffers = 512MB
> 
> which is a bit cumbersome to calculate right now (you'd need = 65536).
> 
> I haven't thought yet how to parse or implement this, but would people find 
> this useful?
> 

It is good idea. I going to implement this. There is some proposal:

Time units is easy:
1h = 60min = 3600s = 3600000ms

Memory units:
What kind of unit prefix will we use for memory?  1kB=1000B and 
1kiBi=1024B  or 1kB=1024kB. See 
http://en.wikipedia.org/wiki/Binary_prefix for detail. I suggest use IEC 
standard convention. By my opinion it is much better.

And there are some other questions:
1) will be unit required? What will be default unit for value without unit?

I suggest mandatory unit avoid the problem with unclear default value.

2) Each internal representation of setting use different unit. Shell I 
convert this representation to milliseconds and bytes? I think it is not 
good idea. It should generate some overflow. I suggest to recompute 
value and round it to integer.

		Zdenek

Re: Units in postgresql.conf

Darcy Buskermolen <darcy@wavefire.com> — 2006-07-20T15:59:05Z

On Thursday 20 July 2006 05:04, Jonah H. Harris wrote:
> On 7/20/06, Peter Eisentraut <peter_e@gmx.net> wrote:
> > I think it would be useful to allow units to be added to these settings,
> > for example
> >
> > shared_buffers = 1000kB
> > checkpoint_warning = 30s
> >
> > This would also allow
> >
> > shared_buffers = 512MB
> >
> > which is a bit cumbersome to calculate right now (you'd need = 65536).
> >
> > I haven't thought yet how to parse or implement this, but would people
> > find this useful?
>
> I agree, a lot of newbies have issues with the configuration file.  I
> have a tiny bit of code (about 20 lines I think) that will handle K,
> M, and G suffixes for memory.  It would be equally easy to add S for
> seconds, ....
>
> In my code, if no suffix existed, I'd just revert to the default
> behavior.  This is probably what we'd want to do in PostgreSQL as
> well.
>
> The only issue in PostgreSQL is knowing what the unit conversion and
> scaling factor is for each parameter (8K, 1K, milliseconds, etc);
> though, this wouldn't be hard to add at all.



Yummy, Yummy, I'd say this would be a big boost in ability to tune for a lot 
of people.

-- 
Darcy Buskermolen
Wavefire Technologies Corp.

http://www.wavefire.com
ph: 250.717.0200
fx: 250.763.1759

Re: Units in postgresql.conf

Josh Berkus <josh@agliodbs.com> — 2006-07-20T16:22:02Z

Peter,

> One frequent source of confusion are the different units that the parameters 
> in postgresql.conf use.  shared_buffers is in 8 kB, work_mem is in 1 kB; 
> bgwriter_delay is in milliseconds, checkpoint_warning is in seconds.  
> Obviously, we can't change that without inconveniencing a lot of users.
> 
> I think it would be useful to allow units to be added to these settings, for 
> example
> 
> shared_buffers = 1000kB
> checkpoint_warning = 30s
> 
> This would also allow
> 
> shared_buffers = 512MB
> 
> which is a bit cumbersome to calculate right now (you'd need = 65536).
> 
> I haven't thought yet how to parse or implement this, but would people find 
> this useful?
> 

Well, it's on my TODO list for 8.2 to write a simple postgresql.conf 
conversion utility in Perl.   If you wanted to make a change like that, 
it would make finishing that mandatory.

Just as well, right now half the vacuum settings are in a different 
section than another half.

--Josh

Re: Units in postgresql.conf

Joe Conway <mail@joeconway.com> — 2006-07-20T16:42:01Z

Peter Eisentraut wrote:
> One frequent source of confusion are the different units that the parameters 
> in postgresql.conf use.  shared_buffers is in 8 kB, work_mem is in 1 kB; 
> bgwriter_delay is in milliseconds, checkpoint_warning is in seconds.  
> Obviously, we can't change that without inconveniencing a lot of users.
> 
> I think it would be useful to allow units to be added to these settings, for 
> example

<snip>
> 
> I haven't thought yet how to parse or implement this, but would people find 
> this useful?

+1

I'd find this useful myself, and I think it would eliminate many 
mistakes by newer admins.

Joe

Re: Units in postgresql.conf

Peter Eisentraut <peter_e@gmx.net> — 2006-07-20T16:49:39Z

Josh Berkus wrote:
> Well, it's on my TODO list for 8.2 to write a simple postgresql.conf
> conversion utility in Perl.   If you wanted to make a change like
> that, it would make finishing that mandatory.

I don't understand how that is related.  Or what a conversion utility 
would be for that matter.

-- 
Peter Eisentraut
http://developer.postgresql.org/~petere/

Re: Units in postgresql.conf

Peter Eisentraut <peter_e@gmx.net> — 2006-07-20T16:51:19Z

Zdenek Kotala wrote:
> Time units is easy:
> 1h = 60min = 3600s = 3600000ms

We don't need anything larger than seconds at the moment.

> What kind of unit prefix will we use for memory?

PostgreSQL has always used 1 kB = 1024 B.

> 1) will be unit required?

No.

> What will be default unit for value without 
> unit?

What we have now.

> I suggest mandatory unit avoid the problem with unclear default
> value.

Not going to happen.

-- 
Peter Eisentraut
http://developer.postgresql.org/~petere/

Re: Units in postgresql.conf

Josh Berkus <josh@agliodbs.com> — 2006-07-20T16:53:20Z

Peter,

> I don't understand how that is related.  Or what a conversion utility
> would be for that matter.

Well, the main issue with changing the units of the PostgreSQL.conf file 
from a user perspective is that the numbers from you 8.0/8.1 conf file 
would no longer work.   A little conversion utilitily to turn your 8.0 
file into an 8.2 file would help solve that.

-- 
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco

Re: Units in postgresql.conf

Peter Eisentraut <peter_e@gmx.net> — 2006-07-20T17:09:13Z

Josh Berkus wrote:
> Well, the main issue with changing the units of the PostgreSQL.conf
> file from a user perspective is that the numbers from you 8.0/8.1
> conf file would no longer work.

No one is intending to do any such change.

-- 
Peter Eisentraut
http://developer.postgresql.org/~petere/

Re: Units in postgresql.conf

Zdenek Kotala <zdenek.kotala@sun.com> — 2006-07-20T20:51:27Z

Peter Eisentraut wrote:
> Zdenek Kotala wrote:
>> Time units is easy:
>> 1h = 60min = 3600s = 3600000ms
> 
> We don't need anything larger than seconds at the moment.
> 
>> What kind of unit prefix will we use for memory?
> 
> PostgreSQL has always used 1 kB = 1024 B.
> 
>> 1) will be unit required?
> 
> No.
> 
>> What will be default unit for value without 
>> unit?
> 
> What we have now.
> 
>> I suggest mandatory unit avoid the problem with unclear default
>> value.
> 
> Not going to happen.
> 

Ok. Conclusion is for time s=second, ms=millisecond and for memory B, 
kB, MB, GB. Unit is not mandatory and if it will missing the behavior 
stays same - backward compatibility (no extra conversion utility).

Last question is if "page" unit should be useful too. For example:

#shared_buffers = 1000                  # min 16 or max_connections*2, 
8KB each

It means 8000kB. But if somebody compiles postgres with different page 
size, than the size will be different. However, somebody should use for 
example 8MB and number of buffers will be 8MB/page_size.

		Zdenek

PS: I have some GUC patches in the patches queue. Could anybody 
test/commit them? I would like continue on the latest version of guc 
subsystem. Thanks

Re: Units in postgresql.conf

Ron Mayer <rm_pg@cheapcomplexdevices.com> — 2006-07-20T22:16:43Z

Peter Eisentraut wrote:
> I think it would be useful to allow units to be added to these settings, for 
> example...
> shared_buffers = 512MB
> which is a bit cumbersome to calculate right now (you'd need = 65536).
> 
> I haven't thought yet how to parse or implement this, but would people find 
> this useful?

Would this extend to things like "random_page_cost" and similar?

If the random_page_cost were specifiable in seconds or ms it might be easier
to someday write a program to measure such values on particular hardware
platforms.   (though I guess for that to work, the config file would also
need to add the reference cost (is it a non-random page access) as well...)

Re: Units in postgresql.conf

Gavin Sherry <swm@linuxworld.com.au> — 2006-07-21T01:05:40Z

On Thu, 20 Jul 2006, Josh Berkus wrote:

> Peter,
>
> > I don't understand how that is related.  Or what a conversion utility
> > would be for that matter.
>
> Well, the main issue with changing the units of the PostgreSQL.conf file
> from a user perspective is that the numbers from you 8.0/8.1 conf file
> would no longer work.   A little conversion utilitily to turn your 8.0
> file into an 8.2 file would help solve that.

Josh,

I would imagine that Peter intends to handle backward compatibility by
processing values without explicit units in the units assumed pre <8.2.

Thanks,

Gavin

Re: Units in postgresql.conf

korryd@enterprisedb.com — 2006-07-21T13:46:08Z

>> Time units is easy:
>> 1h = 60min = 3600s = 3600000ms
>>     
>
> We don't need anything larger than seconds at the moment.
>   
Except for log_rotation_age perhaps?

       -- Korry

Re: Units in postgresql.conf

Josh Berkus <josh@agliodbs.com> — 2006-07-21T17:02:21Z

Gavin, Peter,

> I would imagine that Peter intends to handle backward compatibility by
> processing values without explicit units in the units assumed pre <8.2.

Aha, I misunderstood.

-- 
Josh Berkus
PostgreSQL @ Sun
San Francisco

Re: Units in postgresql.conf

Robert Treat <xzilla@users.sourceforge.net> — 2006-07-22T01:13:07Z

On Thursday 20 July 2006 18:16, Ron Mayer wrote:
> Peter Eisentraut wrote:
> > I think it would be useful to allow units to be added to these settings,
> > for example...
> > shared_buffers = 512MB
> > which is a bit cumbersome to calculate right now (you'd need = 65536).
> >
> > I haven't thought yet how to parse or implement this, but would people
> > find this useful?
>
> Would this extend to things like "random_page_cost" and similar?
>
> If the random_page_cost were specifiable in seconds or ms it might be
> easier to someday write a program to measure such values on particular
> hardware platforms.   (though I guess for that to work, the config file
> would also need to add the reference cost (is it a non-random page access)
> as well...)
>

I'd think no, since random page cost doesn't actually map to any real world 
value.  Unless of course we wanted to add MV for "magic value", but then 
people would want to use that for everything ;-D

-- 
Robert Treat
Build A Brighter LAMP :: Linux Apache {middleware} PostgreSQL