Horton

Thread

  1. Re: What HW / OS is recommeded

    Ben-Nes Michael <miki@canaan.co.il> — 2004-12-16T12:39:26Z

    I think and please correct me that Postgres loves RAM, the more the better.

Any way RAID5 is awful with writing, go with  RAID1 ( mirroring )

I use Debian Sarge and im very happy.

Perl is very slow, maybe you can use PHP ?


Alex wrote:

> Hi,
> we are planning to upgrade our servers but deciding on the right 
> configuration seems to be quite difficult.
>
> As for the system. About 50 tables, 20M records and growing about 
> 500k-1m per month.
> The systems mostly loads data from files (perl batch jobs). And 
> generates client files. Jobs generally dont run at the same time, but 
> timely loading/delivery is very important. We also run tomcat with no 
> more than 5-10 concurrent users connecting. (mostly browsing data)
>
> We are currently looking at Dell / HP
> but the questions is
>
> - how many processors (2 or 4)
> - do we gain with 4 cpus if we probably never have a few users connected
> - what processors are recommended  Opteron / Xeon / Itanium
> - how much memory ?  2GB ? 4GB ?
> - Disks, i guess we go with Raid5, 15k SCSI
> - what OS ? Suse / RHE3 / Fedora /
> - Disk controller ?
>
>
> Currently we run it on a Dell Blade, dual P3 1.4ghz with 1G memory
> Adding 1GB memory did actually not bring much performance gains.
>
> Does anyone have some first hand experience? Can anybody point me to 
> some resources ? Or recommend certain systems?
> What kind of performance gain can be expected going from a P3 to a 
> higher end processor ?
> Thanks for any suggestions
>
> Alex
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
>               http://www.postgresql.org/docs/faqs/FAQ.html


-- 
--------------------------
Canaan Surfing Ltd.
Internet Service Providers
Ben-Nes Michael - Manager
Tel: 972-4-6991122
Cel: 972-52-8555757
Fax: 972-4-6990098
http://www.canaan.net.il
--------------------------

  2. What HW / OS is recommeded

    Alex <alex@meerkatsoft.com> — 2004-12-16T13:12:05Z

    Hi,
we are planning to upgrade our servers but deciding on the right 
configuration seems to be quite difficult.

As for the system. About 50 tables, 20M records and growing about 
500k-1m per month.
The systems mostly loads data from files (perl batch jobs). And 
generates client files. Jobs generally dont run at the same time, but 
timely loading/delivery is very important. We also run tomcat with no 
more than 5-10 concurrent users connecting. (mostly browsing data)

We are currently looking at Dell / HP
but the questions is

- how many processors (2 or 4)
- do we gain with 4 cpus if we probably never have a few users connected
- what processors are recommended  Opteron / Xeon / Itanium
- how much memory ?  2GB ? 4GB ?
- Disks, i guess we go with Raid5, 15k SCSI
- what OS ? Suse / RHE3 / Fedora /
- Disk controller ?


Currently we run it on a Dell Blade, dual P3 1.4ghz with 1G memory
Adding 1GB memory did actually not bring much performance gains.

Does anyone have some first hand experience? Can anybody point me to 
some resources ? Or recommend certain systems?
What kind of performance gain can be expected going from a P3 to a 
higher end processor ? 

Thanks for any suggestions

Alex

  3. Re: What HW / OS is recommeded

    Geoffrey <esoteric@3times25.net> — 2004-12-16T13:59:06Z

    Michael Ben-Nes wrote:
> I think and please correct me that Postgres loves RAM, the more the better.
> 
> Any way RAID5 is awful with writing, go with  RAID1 ( mirroring )
> 
> I use Debian Sarge and im very happy.
> 
> Perl is very slow, maybe you can use PHP ?

I find perl perfectly acceptable.  I would appreciate some 
examples/benchmarks/comparisons?

-- 
Until later, Geoffrey

  4. Re: What HW / OS is recommeded

    Richard Huxton <dev@archonet.com> — 2004-12-16T13:59:24Z

    Michael Ben-Nes wrote:
> I think and please correct me that Postgres loves RAM, the more the better.

Certainly for disk-cache.

> Any way RAID5 is awful with writing, go with  RAID1 ( mirroring )

Raid 10 seems to be the consensus if you have enough disks. See the 
archives of the performance list for plenty of discussion.

> I use Debian Sarge and im very happy.

Not sure it makes much difference performance-wise. I'd use whatever 
flavour of Linux you have the most experience with.

> Perl is very slow, maybe you can use PHP ?

If your Perl is slower than your PHP, you need to get a better Perl 
programmer ;-)
In any case, if your application is too slow for your database then all 
the previous tuning is largely irrelevant.

--
   Richard Huxton
   Archonet Ltd

  5. Re: What HW / OS is recommeded

    Alex <alex@meerkatsoft.com> — 2004-12-16T14:48:22Z

    Hmm...
I read that Raid5 is suggested over Raid1. Also HW vendors told us that.
Php :-) is not an option and I dont believe Perl is a bottleneck as well.



Michael Ben-Nes wrote:

> I think and please correct me that Postgres loves RAM, the more the 
> better.
>
> Any way RAID5 is awful with writing, go with  RAID1 ( mirroring )
>
> I use Debian Sarge and im very happy.
>
> Perl is very slow, maybe you can use PHP ?
>
>
> Alex wrote:
>
>> Hi,
>> we are planning to upgrade our servers but deciding on the right 
>> configuration seems to be quite difficult.
>>
>> As for the system. About 50 tables, 20M records and growing about 
>> 500k-1m per month.
>> The systems mostly loads data from files (perl batch jobs). And 
>> generates client files. Jobs generally dont run at the same time, but 
>> timely loading/delivery is very important. We also run tomcat with no 
>> more than 5-10 concurrent users connecting. (mostly browsing data)
>>
>> We are currently looking at Dell / HP
>> but the questions is
>>
>> - how many processors (2 or 4)
>> - do we gain with 4 cpus if we probably never have a few users connected
>> - what processors are recommended  Opteron / Xeon / Itanium
>> - how much memory ?  2GB ? 4GB ?
>> - Disks, i guess we go with Raid5, 15k SCSI
>> - what OS ? Suse / RHE3 / Fedora /
>> - Disk controller ?
>>
>>
>> Currently we run it on a Dell Blade, dual P3 1.4ghz with 1G memory
>> Adding 1GB memory did actually not bring much performance gains.
>>
>> Does anyone have some first hand experience? Can anybody point me to 
>> some resources ? Or recommend certain systems?
>> What kind of performance gain can be expected going from a P3 to a 
>> higher end processor ?
>> Thanks for any suggestions
>>
>> Alex
>>
>>
>> ---------------------------(end of broadcast)---------------------------
>> TIP 5: Have you checked our extensive FAQ?
>>
>>               http://www.postgresql.org/docs/faqs/FAQ.html
>
>
>

  6. Re: What HW / OS is recommeded

    Scott Marlowe <smarlowe@g2switchworks.com> — 2004-12-16T15:17:00Z

    On Thu, 2004-12-16 at 06:39, Michael Ben-Nes wrote:
> I think and please correct me that Postgres loves RAM, the more the better.
> 
> Any way RAID5 is awful with writing, go with  RAID1 ( mirroring )

With battery backed cache and a large array, RAID 5 is quite fast, even
with writes.  Plus with a lot of drives in a mostly read environment,
it's quite likely that each read will hit a different drive so that many
parallel requests can be handled quite well.  The general rule I use is
6 or fewer drives will do better in RAID 1+0, 7 or more will tend to do
better with RAID 5.

> Perl is very slow, maybe you can use PHP ?

While mod_perl and its relations have never been fast running under
apache in comparison to PHP, it's no slouch, paying mostly in startup
time, not run time.  For complex apps, the startup time difference
becomes noise compared to the run time, so it's no big advantage to
PHP.  I really like PHP by the way.  But Perl is pretty nice too. 

Run the Unix OS you're most comfortable with, knowing that PostgreSQL
gets lots of testing on the free unixes more so than on the commercial
ones.  Give it a machine with plenty of RAM and a fast I/O subsystem,
and two CPUS and you'll get good performance.  If your needs exceed the
performance of one of these machines, you're probably better off going
to a pgpool / slony cluster than trying to build a bigger machine.

  7. Re: What HW / OS is recommeded

    Geoffrey <esoteric@3times25.net> — 2004-12-16T15:18:28Z

    Alex wrote:
> Hmm...
> I read that Raid5 is suggested over Raid1. Also HW vendors told us that.
> Php :-) is not an option and I dont believe Perl is a bottleneck as well.

Why would your HW vendor be stipulating the software you use?

-- 
Until later, Geoffrey

  8. Re: What HW / OS is recommeded

    Keith C. Perry <netadmin@vcsn.com> — 2004-12-16T16:57:02Z

    Quoting Scott Marlowe <smarlowe@g2switchworks.com>:

> On Thu, 2004-12-16 at 06:39, Michael Ben-Nes wrote:
> > I think and please correct me that Postgres loves RAM, the more the
> better.
> > 
> > Any way RAID5 is awful with writing, go with  RAID1 ( mirroring )
> 
> With battery backed cache and a large array, RAID 5 is quite fast, even
> with writes.  Plus with a lot of drives in a mostly read environment,
> it's quite likely that each read will hit a different drive so that many
> parallel requests can be handled quite well.  The general rule I use is
> 6 or fewer drives will do better in RAID 1+0, 7 or more will tend to do
> better with RAID 5.
> 
> > Perl is very slow, maybe you can use PHP ?
> 
> While mod_perl and its relations have never been fast running under
> apache in comparison to PHP, it's no slouch, paying mostly in startup
> time, not run time.  For complex apps, the startup time difference
> becomes noise compared to the run time, so it's no big advantage to
> PHP.  I really like PHP by the way.  But Perl is pretty nice too. 

I run apache2, ssl, mod_perl and php.  I have yet to hear complaints from my
perl or php programmer.  Without have another PHP vs. Perl "thing" lets all
agree that they are both pretty nice  :)  

> Run the Unix OS you're most comfortable with, knowing that PostgreSQL
> gets lots of testing on the free unixes more so than on the commercial
> ones.  Give it a machine with plenty of RAM and a fast I/O subsystem,
> and two CPUS and you'll get good performance.  If your needs exceed the
> performance of one of these machines, you're probably better off going
> to a pgpool / slony cluster than trying to build a bigger machine.

I'm not sure I heard any mention of filesystems but I've been moving all my EXT3
filesystems to XFS.  Some other journaling filesystem that you might want to
look into are JFS and ReiserFS.

-- 
Keith C. Perry, MS E.E.
Director of Networks & Applications
VCSN, Inc.
http://vcsn.com
 
____________________________________
This email account is being host by:
VCSN, Inc : http://vcsn.com

  9. Re: What HW / OS is recommeded

    Alex <alex@meerkatsoft.com> — 2004-12-17T01:04:42Z

    The comment about HW vendor was regarding Raid configuration not the 
software.


Geoffrey wrote:

> Alex wrote:
>
>> Hmm...
>> I read that Raid5 is suggested over Raid1. Also HW vendors told us that.
>> Php :-) is not an option and I dont believe Perl is a bottleneck as 
>> well.
>
>
> Why would your HW vendor be stipulating the software you use?
>

  10. Re: What HW / OS is recommeded

    Alex <alex@meerkatsoft.com> — 2004-12-17T01:10:25Z

    We use perl for the heavy batch jobs, the web interface is written using 
JSP / applets.
If we would change these then it would be Java or C. But all the heavy 
stuff is handled by Stored Procedures so  I dont see a real need for a 
change.

I actually am more interested to hear if there are an recommended 
systems or setups.
Also with regard to 2/4 CPUs or 32/64 bit etc.





Scott Marlowe wrote:

>On Thu, 2004-12-16 at 06:39, Michael Ben-Nes wrote:
>  
>
>>I think and please correct me that Postgres loves RAM, the more the better.
>>
>>Any way RAID5 is awful with writing, go with  RAID1 ( mirroring )
>>    
>>
>
>With battery backed cache and a large array, RAID 5 is quite fast, even
>with writes.  Plus with a lot of drives in a mostly read environment,
>it's quite likely that each read will hit a different drive so that many
>parallel requests can be handled quite well.  The general rule I use is
>6 or fewer drives will do better in RAID 1+0, 7 or more will tend to do
>better with RAID 5.
>
>  
>
>>Perl is very slow, maybe you can use PHP ?
>>    
>>
>
>While mod_perl and its relations have never been fast running under
>apache in comparison to PHP, it's no slouch, paying mostly in startup
>time, not run time.  For complex apps, the startup time difference
>becomes noise compared to the run time, so it's no big advantage to
>PHP.  I really like PHP by the way.  But Perl is pretty nice too. 
>
>Run the Unix OS you're most comfortable with, knowing that PostgreSQL
>gets lots of testing on the free unixes more so than on the commercial
>ones.  Give it a machine with plenty of RAM and a fast I/O subsystem,
>and two CPUS and you'll get good performance.  If your needs exceed the
>performance of one of these machines, you're probably better off going
>to a pgpool / slony cluster than trying to build a bigger machine.
>
>
>  
>

  11. Re: What HW / OS is recommeded

    Geoffrey <esoteric@3times25.net> — 2004-12-17T03:59:00Z

    Alex wrote:
> The comment about HW vendor was regarding Raid configuration not the 
> software.

My apologies, misread your post.

> 
> 
> Geoffrey wrote:
> 
>> Alex wrote:
>>
>>> Hmm...
>>> I read that Raid5 is suggested over Raid1. Also HW vendors told us that.
>>> Php :-) is not an option and I dont believe Perl is a bottleneck as 
>>> well.
>>
>>
>>
>> Why would your HW vendor be stipulating the software you use?



-- 
Until later, Geoffrey

  12. Re: What HW / OS is recommeded

    Scott Marlowe <smarlowe@g2switchworks.com> — 2004-12-17T14:15:26Z

    On Thu, 2004-12-16 at 19:10, Alex wrote:

> I actually am more interested to hear if there are an recommended 
> systems or setups.
> Also with regard to 2/4 CPUs or 32/64 bit etc.

Sorry to have gotten off on a tangent there.  Posts in the last year or
so to the -performance mailing list have shown the 64 bit AMD platform
to be the fastest X86 based platform around, and having 64 bit hardware
is nice for postgresql installations dealing with large data sets.

Generally, 2 CPUs is plenty.  The fastest storage systems seems to be 
SAN based, with large local RAID arrays with battery backed cache coming
in a close second.  Once the RAID array or SAN device has enough drives,
putting things like the transaction log elsewhere have little effect.

But it's really all about what you're doing with your database.  If
you're taking huge data sets and running statistical analysis with the
plR, you'll need lots of memory as well as plenty of CPU horsepower.  If
you're handling thousands of simultaneous air line reservations, you'll
need lots of drives, and a fair number of CPUs, but probably not so much
memory.  You may need clusters for one solution, but find one big server
is the answer to another problem.

There's no one simple answer, because there's no one simple problem.

But for most use cases, a dual AMD 64 bit CPU, 4 gigs of ram, and a half
a dozen hard drives is a good starting point.

  13. Re: What HW / OS is recommeded

    Vivek Khera <khera@kcilink.com> — 2004-12-22T17:28:22Z

    >>>>> "a" == alex  <alex@meerkatsoft.com> writes:


a> We are currently looking at Dell / HP
a> but the questions is

a> - how many processors (2 or 4)
a> - do we gain with 4 cpus if we probably never have a few users connected
a> - what processors are recommended  Opteron / Xeon / Itanium
a> - how much memory ?  2GB ? 4GB ?
a> - Disks, i guess we go with Raid5, 15k SCSI
a> - what OS ? Suse / RHE3 / Fedora /
a> - Disk controller ?


Run, do not walk, from your Dell solution.  I've never been able to
get "expected" performance from those boxes.  They seem to do
something to the RAID controllers to make them not work as fast as one
would expect the equivalent name-brand part (eg, LSI RAID card or
Adaptec RAID card) and similar disk drives.

Others have commented on the other aspects.  The RAID configuration is
something everyone has an opinion for.  I personally run a RAID5 on 14
disks, but the new server replacing this Dell box will have RAID10 on
6 disks for data and RAID1 on 2 disks for pg_xlog + system.  It is a
dual Opteron with 4GB RAM and an Adaptec controller.  I only run
FreeBSD.

See the performance mailing list for a big discussion on vendors just
two or three weeks ago.


-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Vivek Khera, Ph.D.                Khera Communications, Inc.
Internet: khera@kciLink.com       Rockville, MD  +1-301-869-4449 x806
AIM: vivekkhera Y!: vivek_khera   http://www.khera.org/~vivek/

  14. Re: What HW / OS is recommeded

    Lonni J Friedman <netllama@gmail.com> — 2004-12-22T17:41:37Z

    On Wed, 22 Dec 2004 12:28:22 -0500, Vivek Khera <khera@kcilink.com> wrote:
> >>>>> "a" == alex  <alex@meerkatsoft.com> writes:
> 
> a> We are currently looking at Dell / HP
> a> but the questions is
> 
> a> - how many processors (2 or 4)
> a> - do we gain with 4 cpus if we probably never have a few users connected
> a> - what processors are recommended  Opteron / Xeon / Itanium
> a> - how much memory ?  2GB ? 4GB ?
> a> - Disks, i guess we go with Raid5, 15k SCSI
> a> - what OS ? Suse / RHE3 / Fedora /
> a> - Disk controller ?
> 
> Run, do not walk, from your Dell solution.  I've never been able to
> get "expected" performance from those boxes.  They seem to do
> something to the RAID controllers to make them not work as fast as one
> would expect the equivalent name-brand part (eg, LSI RAID card or
> Adaptec RAID card) and similar disk drives.

Hate to burst your bubble, but the RAID controller that Dell ships is
an Adaptec OEM.  Dell just rebrands them.


-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
L. Friedman                                    netllama@gmail.com
LlamaLand                       http://netllama.linux-sxs.org

  15. Re: What HW / OS is recommeded

    Scott Marlowe <smarlowe@g2switchworks.com> — 2004-12-22T18:09:43Z

    On Wed, 2004-12-22 at 11:41, Lonni J Friedman wrote:
> On Wed, 22 Dec 2004 12:28:22 -0500, Vivek Khera <khera@kcilink.com> wrote:
> > >>>>> "a" == alex  <alex@meerkatsoft.com> writes:
> > 
> > a> We are currently looking at Dell / HP
> > a> but the questions is
> > 
> > a> - how many processors (2 or 4)
> > a> - do we gain with 4 cpus if we probably never have a few users connected
> > a> - what processors are recommended  Opteron / Xeon / Itanium
> > a> - how much memory ?  2GB ? 4GB ?
> > a> - Disks, i guess we go with Raid5, 15k SCSI
> > a> - what OS ? Suse / RHE3 / Fedora /
> > a> - Disk controller ?
> > 
> > Run, do not walk, from your Dell solution.  I've never been able to
> > get "expected" performance from those boxes.  They seem to do
> > something to the RAID controllers to make them not work as fast as one
> > would expect the equivalent name-brand part (eg, LSI RAID card or
> > Adaptec RAID card) and similar disk drives.
> 
> Hate to burst your bubble, but the RAID controller that Dell ships is
> an Adaptec OEM.  Dell just rebrands them.

I've use the Dell PERC 4DC and had VERY good performance from it.  IT's
the late model U320 LSI MegaRAID and runs great.  I do remember that the
2650 and few other Dells had the serverworks chipset in them that caused
a lot of context switches in heavy parallel load in a discussion on the
performance list.  We weren't running heavy parallel, just a report
server with a dozen or so users, so it wasn't an issue for us.

  16. Re: What HW / OS is recommeded

    Vivek Khera <khera@kcilink.com> — 2004-12-22T18:44:11Z

    On Dec 22, 2004, at 12:41 PM, Lonni J Friedman wrote:

>> get "expected" performance from those boxes.  They seem to do
>> something to the RAID controllers to make them not work as fast as one
>> would expect the equivalent name-brand part (eg, LSI RAID card or
>> Adaptec RAID card) and similar disk drives.
>
> Hate to burst your bubble, but the RAID controller that Dell ships is
> an Adaptec OEM.  Dell just rebrands them.

They also rebrand LSI.  I'm saying that the versions they sell don't 
seem to perform as well as the manufacturer branded ones from what I've 
compared with other folks.

Vivek Khera, Ph.D.
+1-301-869-4449 x806

  17. Re: What HW / OS is recommeded

    Vivek Khera <khera@kcilink.com> — 2004-12-22T18:46:21Z

    On Dec 22, 2004, at 1:09 PM, Scott Marlowe wrote:

> I've use the Dell PERC 4DC and had VERY good performance from it.  IT's
> the late model U320 LSI MegaRAID and runs great.  I do remember that 
> the
> 2650 and few other Dells had the serverworks chipset in them that 
> caused
> a lot of context switches in heavy parallel load in a discussion on the
> performance list.  We weren't running heavy parallel, just a report
> server with a dozen or so users, so it wasn't an issue for us.
>

Whatever the ultimate cause, I can't say. But I can say that I'm not 
planning to buy any more Dell servers for my database needs.  Both my 
2450 and 2650 basically suck at I/O load with any kind of concurrent 
access.  I might as well be running IDE :-(

  18. Re: What HW / OS is recommeded

    Geoffrey <esoteric@3times25.net> — 2004-12-23T00:55:05Z

    Lonni J Friedman wrote:
> On Wed, 22 Dec 2004 12:28:22 -0500, Vivek Khera <khera@kcilink.com> wrote:
> 
>>>>>>>"a" == alex  <alex@meerkatsoft.com> writes:
>>
>>a> We are currently looking at Dell / HP
>>a> but the questions is
>>
>>a> - how many processors (2 or 4)
>>a> - do we gain with 4 cpus if we probably never have a few users connected
>>a> - what processors are recommended  Opteron / Xeon / Itanium
>>a> - how much memory ?  2GB ? 4GB ?
>>a> - Disks, i guess we go with Raid5, 15k SCSI
>>a> - what OS ? Suse / RHE3 / Fedora /
>>a> - Disk controller ?
>>
>>Run, do not walk, from your Dell solution.  I've never been able to
>>get "expected" performance from those boxes.  They seem to do
>>something to the RAID controllers to make them not work as fast as one
>>would expect the equivalent name-brand part (eg, LSI RAID card or
>>Adaptec RAID card) and similar disk drives.
> 
> 
> Hate to burst your bubble, but the RAID controller that Dell ships is
> an Adaptec OEM.  Dell just rebrands them.

Dell is known for having their own 'version' of hardware made, 
specifically to reduce the cost of the machines.  I don't know that this 
is the case with this adapter, but I would not ever purchase another 
dell machine.

-- 
Until later, Geoffrey