Thread

Pg+Linux swap use

alexandre :: aldeia digital <alepaes@aldeiadigital.com.br> — 2003-10-30T19:49:08Z

Hi,

Old: Post 7.3.2, P4 1.8, 1 MB RAM, 2 x IDE SW RAID 1, RedHat 8
New: Post 7.3.4, Xeon 2.4, 1 MB RAM, 2 x SCSI 15k SW RAID 1, RedHat 9

Both use: Only postgresql on server. Buffers = 8192, effective cache = 100000

In old plataform the free and vmstat reports no use of swap.
In new, the swap is in constant use (40-100 MB), with a low but constant
swap out and swap in. The cache memory ~ 830000 and buffers ~ 43000

I try to reduce the buffers to 1024 with no effects.

Thanks,

Alexandre

Re: Pg+Linux swap use

scott.marlowe <scott.marlowe@ihs.com> — 2003-10-30T20:08:50Z

On Thu, 30 Oct 2003, alexandre :: aldeia digital wrote:

> Hi,
> 
> Old: Post 7.3.2, P4 1.8, 1 MB RAM, 2 x IDE SW RAID 1, RedHat 8
> New: Post 7.3.4, Xeon 2.4, 1 MB RAM, 2 x SCSI 15k SW RAID 1, RedHat 9
> 
> Both use: Only postgresql on server. Buffers = 8192, effective cache = 100000
> 
> In old plataform the free and vmstat reports no use of swap.
> In new, the swap is in constant use (40-100 MB), with a low but constant
> swap out and swap in. The cache memory ~ 830000 and buffers ~ 43000

Do both machines have the same amount of swap, and what kernels are the 
two running?  Also, do the xeons have hyperthreading turned on.  I doubt 
hyperthreading has anything to do with this problem, but I've found that 
turning off hyperthreading in the BIOS gets me a faster machine under 
Postgresql so I just thought I'd throw that out there.

Re: Pg+Linux swap use

Jeff Trout <threshar@torgo.978.org> — 2003-10-31T12:41:14Z

On Thu, 30 Oct 2003 17:49:08 -0200 (BRST)
"alexandre :: aldeia digital" <alepaes@aldeiadigital.com.br> wrote:

> Both use: Only postgresql on server. Buffers = 8192, effective cache =
> 100000
> 

Well, I'm assuming you meant 1GB of ram, not 1MB :)

Check a ps auxw to see what is running. Perhaps X is running gobbling up
your precious mem.  But still.. with 1GB there should be virtually no
swap activity.  

How busy is the DB? How many connections?

and is sort_mem set high?

-- 
Jeff Trout <jeff@jefftrout.com>
http://www.jefftrout.com/
http://www.stuarthamm.net/

Re: Pg+Linux swap use

Shridhar Daithankar <shridhar_daithankar@persistent.co.in> — 2003-10-31T12:52:07Z

Jeff wrote:

> On Thu, 30 Oct 2003 17:49:08 -0200 (BRST)
> "alexandre :: aldeia digital" <alepaes@aldeiadigital.com.br> wrote:
> 
> 
>>Both use: Only postgresql on server. Buffers = 8192, effective cache =
>>100000
> Well, I'm assuming you meant 1GB of ram, not 1MB :)
> 
> Check a ps auxw to see what is running. Perhaps X is running gobbling up
> your precious mem.  But still.. with 1GB there should be virtually no
> swap activity.  
> 
> How busy is the DB? How many connections?
> 
> and is sort_mem set high?

Also are two kernels exactly same? In my experience linux kernel behaves 
slightly different from version to version w.r.t swap aggressiveness...

  Shridhar

Re: Pg+Linux swap use

alexandre :: aldeia digital <alepaes@aldeiadigital.com.br> — 2003-10-31T14:03:59Z

Scott, Jeff and Shridhar:

1 GB RAM :)

The stock kernels are not the same, HyperThreading enabled. 80
simultaneous connections. sort_mem = 4096

I will compile my own kernel on this weekend, and I will report
to the list after.

Thank's all

Alexandre


> Also are two kernels exactly same? In my experience linux kernel behaves
> slightly different from version to version w.r.t swap aggressiveness...
>
>   Shridhar
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
>

Re: Pg+Linux swap use

Andrew Sullivan <andrew@libertyrms.info> — 2003-10-31T14:36:24Z

On Fri, Oct 31, 2003 at 12:03:59PM -0200, alexandre :: aldeia digital wrote:
> Scott, Jeff and Shridhar:
> 
> 1 GB RAM :)
> 
> The stock kernels are not the same, HyperThreading enabled. 80

Some people have reported that things actually slow down with HT
enabled.  Have you tried turning it off?

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Afilias Canada                        Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110

Re: Pg+Linux swap use

Rob Sell <lists@facnd.com> — 2003-10-31T15:31:19Z

Not being one to hijack threads, but I haven't heard of this performance hit
when using HT, I have what should all rights be a pretty fast server, dual
2.4 Xeons with HT 205gb raid 5 array, 1 gig of memory. And it is only 50% as
fast as my old server which was a dual AMD MP 1400's with a 45gb raid 5
array and 1gb of ram. I have read everything I could find on Pg performance
tweaked all the variables that were suggested and nothing. Which is why I
subscribed to this list, just been lurking so far but this caught my eye. 

Rob

-----Original Message-----
From: pgsql-performance-owner@postgresql.org
[mailto:pgsql-performance-owner@postgresql.org] On Behalf Of Andrew Sullivan
Sent: Friday, October 31, 2003 8:36 AM
To: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Pg+Linux swap use

On Fri, Oct 31, 2003 at 12:03:59PM -0200, alexandre :: aldeia digital wrote:
> Scott, Jeff and Shridhar:
> 
> 1 GB RAM :)
> 
> The stock kernels are not the same, HyperThreading enabled. 80

Some people have reported that things actually slow down with HT
enabled.  Have you tried turning it off?

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Afilias Canada                        Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

Re: Pg+Linux swap use

Jeff Trout <threshar@torgo.978.org> — 2003-10-31T15:39:02Z

On Fri, 31 Oct 2003 09:31:19 -0600
"Rob Sell" <lists@facnd.com> wrote:

> Not being one to hijack threads, but I haven't heard of this
> performance hit when using HT, I have what should all rights be a
> pretty fast server, dual 2.4 Xeons with HT 205gb raid 5 array, 1 gig
> of memory. And it is only 50% as fast as my old server which was a
> dual AMD MP 1400's with a 45gb raid 5 array and 1gb of ram. I have
> read everything I could find on Pg performance tweaked all the
> variables that were suggested and nothing. Which is why I subscribed
> to this list, just been lurking so far but this caught my eye. 
> 
> Rob

There's benchmarks around that show in _some_ cases HT is not all it is
cracked up to be, somtimes running slower.

But the real point of this thread isn't all this stuff about
hyperthreading, the problem is the guy is seeing swapping..

I'm guessing RH is running some useless stuff in the BG.. or maybe he's
running a retarded kernel... or.. maybe.. just.. maybe.. little elves
are doing it.

-- 
Jeff Trout <jeff@jefftrout.com>
http://www.jefftrout.com/
http://www.stuarthamm.net/

Re: Pg+Linux swap use

Shridhar Daithankar <shridhar_daithankar@persistent.co.in> — 2003-10-31T15:54:43Z

Jeff wrote:

> On Fri, 31 Oct 2003 09:31:19 -0600
> "Rob Sell" <lists@facnd.com> wrote:
> 
> 
>>Not being one to hijack threads, but I haven't heard of this
>>performance hit when using HT, I have what should all rights be a
>>pretty fast server, dual 2.4 Xeons with HT 205gb raid 5 array, 1 gig
>>of memory. And it is only 50% as fast as my old server which was a
>>dual AMD MP 1400's with a 45gb raid 5 array and 1gb of ram. I have
>>read everything I could find on Pg performance tweaked all the
>>variables that were suggested and nothing. Which is why I subscribed
>>to this list, just been lurking so far but this caught my eye. 
>>
>>Rob
> There's benchmarks around that show in _some_ cases HT is not all it is
> cracked up to be, somtimes running slower.

To use HT effectively on needs.

1. A kernel that understands HT.
2. A task scheduler that understands HT
3. A CPU intensive load.

So if you are running a stock RedHat and production postgresql database, turn it 
off. It won't hurt certainly(Almost certainly)

> I'm guessing RH is running some useless stuff in the BG.. or maybe he's
> running a retarded kernel... or.. maybe.. just.. maybe.. little elves
> are doing it.

Too much..:-)

I guess Alexandre can tune bdflush to be little less agressive. Comparing 
bdflush values on two machines might turn up something.

His idea of compiling kernel is also good one. He can also try tuning some 
values in /proc/sys/vm but I don't find any documentation offhand.

I usually run slackware and a handcompiled 2.6-test4. None of them use any swap 
unless true memory starts falling low. This 
touch-swap-even-if-oodles-of-ram-is-free is something I have't experienced on my 
desktop for quite a while.

  Shridhar

Re: Pg+Linux swap use

PT <wmoran@potentialtech.com> — 2003-10-31T15:55:33Z

Just for an additional viewpoint.  I'm finishing up a project based on FreeBSD
and PostgreSQL.  The target server is a Dual 2.4G Intel machine.  I have tested
the application with hyperthreading enabled and disabled.  To all appearances,
enabling hyperthreading makes the box act like a quad, with the expected increase
in processing capability - _for_this_application_.

I have also heard the claims and seen the tests that show hyperthreading
occasionally decreasing performance.  I think in the end, you just have to
test your particular application to see how it reacts.

Rob Sell wrote:
> Not being one to hijack threads, but I haven't heard of this performance hit
> when using HT, I have what should all rights be a pretty fast server, dual
> 2.4 Xeons with HT 205gb raid 5 array, 1 gig of memory. And it is only 50% as
> fast as my old server which was a dual AMD MP 1400's with a 45gb raid 5
> array and 1gb of ram. I have read everything I could find on Pg performance
> tweaked all the variables that were suggested and nothing. Which is why I
> subscribed to this list, just been lurking so far but this caught my eye. 
> 
> Rob
> 
> -----Original Message-----
> From: pgsql-performance-owner@postgresql.org
> [mailto:pgsql-performance-owner@postgresql.org] On Behalf Of Andrew Sullivan
> Sent: Friday, October 31, 2003 8:36 AM
> To: pgsql-performance@postgresql.org
> Subject: Re: [PERFORM] Pg+Linux swap use
> 
> On Fri, Oct 31, 2003 at 12:03:59PM -0200, alexandre :: aldeia digital wrote:
> 
>>Scott, Jeff and Shridhar:
>>
>>1 GB RAM :)
>>
>>The stock kernels are not the same, HyperThreading enabled. 80
> 
> 
> Some people have reported that things actually slow down with HT
> enabled.  Have you tried turning it off?
> 
> A
> 

-- 
Bill Moran
Potential Technologies
http://www.potentialtech.com

Re: Pg+Linux swap use

Greg Stark <gsstark@mit.edu> — 2003-10-31T16:37:45Z

Bill Moran <wmoran@potentialtech.com> writes:

> Just for an additional viewpoint.  I'm finishing up a project based on FreeBSD
> and PostgreSQL.  The target server is a Dual 2.4G Intel machine.  I have tested
> the application with hyperthreading enabled and disabled.  To all appearances,
> enabling hyperthreading makes the box act like a quad, with the expected increase
> in processing capability - _for_this_application_.
> 
> I have also heard the claims and seen the tests that show hyperthreading
> occasionally decreasing performance.  I think in the end, you just have to
> test your particular application to see how it reacts.

My understanding is that the case where HT hurts is precisely your case. When
you have two real processors with HT the kernel will sometimes schedule two
jobs on the two virtual processors on the same real processor leaving the two
virtual processors on the other real processor idle.

As far as I know a single processor machine with HT does not benefit from
disabling HT.

-- 
greg

Re: Pg+Linux swap use

Rob Sell <lists@facnd.com> — 2003-10-31T16:42:06Z

For the record I am running on SuSE with a pretty much stock kernel. Not to
sound naïve, but is turning of HT something done in the bios?

Rob

-----Original Message-----
From: pgsql-performance-owner@postgresql.org
[mailto:pgsql-performance-owner@postgresql.org] On Behalf Of Bill Moran
Sent: Friday, October 31, 2003 9:56 AM
To: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Pg+Linux swap use

Just for an additional viewpoint.  I'm finishing up a project based on
FreeBSD
and PostgreSQL.  The target server is a Dual 2.4G Intel machine.  I have
tested
the application with hyperthreading enabled and disabled.  To all
appearances,
enabling hyperthreading makes the box act like a quad, with the expected
increase
in processing capability - _for_this_application_.

I have also heard the claims and seen the tests that show hyperthreading
occasionally decreasing performance.  I think in the end, you just have to
test your particular application to see how it reacts.

Rob Sell wrote:
> Not being one to hijack threads, but I haven't heard of this performance
hit
> when using HT, I have what should all rights be a pretty fast server, dual
> 2.4 Xeons with HT 205gb raid 5 array, 1 gig of memory. And it is only 50%
as
> fast as my old server which was a dual AMD MP 1400's with a 45gb raid 5
> array and 1gb of ram. I have read everything I could find on Pg
performance
> tweaked all the variables that were suggested and nothing. Which is why I
> subscribed to this list, just been lurking so far but this caught my eye. 
> 
> Rob
> 
> -----Original Message-----
> From: pgsql-performance-owner@postgresql.org
> [mailto:pgsql-performance-owner@postgresql.org] On Behalf Of Andrew
Sullivan
> Sent: Friday, October 31, 2003 8:36 AM
> To: pgsql-performance@postgresql.org
> Subject: Re: [PERFORM] Pg+Linux swap use
> 
> On Fri, Oct 31, 2003 at 12:03:59PM -0200, alexandre :: aldeia digital
wrote:
> 
>>Scott, Jeff and Shridhar:
>>
>>1 GB RAM :)
>>
>>The stock kernels are not the same, HyperThreading enabled. 80
> 
> 
> Some people have reported that things actually slow down with HT
> enabled.  Have you tried turning it off?
> 
> A
> 


-- 
Bill Moran
Potential Technologies
http://www.potentialtech.com


---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

               http://www.postgresql.org/docs/faqs/FAQ.html

Re: Pg+Linux swap use

PT <wmoran@potentialtech.com> — 2003-10-31T16:55:49Z

Greg Stark wrote:
> Bill Moran <wmoran@potentialtech.com> writes:
> 
>>Just for an additional viewpoint.  I'm finishing up a project based on FreeBSD
>>and PostgreSQL.  The target server is a Dual 2.4G Intel machine.  I have tested
>>the application with hyperthreading enabled and disabled.  To all appearances,
>>enabling hyperthreading makes the box act like a quad, with the expected increase
>>in processing capability - _for_this_application_.
>>
>>I have also heard the claims and seen the tests that show hyperthreading
>>occasionally decreasing performance.  I think in the end, you just have to
>>test your particular application to see how it reacts.
> 
> My understanding is that the case where HT hurts is precisely your case. When
> you have two real processors with HT the kernel will sometimes schedule two
> jobs on the two virtual processors on the same real processor leaving the two
> virtual processors on the other real processor idle.

Yup, that's why I tested it.

While more testing is probably in order, I could find no disadvantages to running
with HTT enabled.  And when I ran many background jobs (which is likely to happen
during batch processing on this sytem) the system seemed to perform as if it
really had 4 processors.

Perhaps this is an indication of the quality of the FreeBSD scheduler (maybe it's
HTT aware and makes decisions accordingly?), but I'm not involved enough in that
level of development to do any more than speculate.

Again, this is pure speculation ... can anyone with a more technical insight
comment on whether my guess is correct, or whether I'm not testing enough?

-- 
Bill Moran
Potential Technologies
http://www.potentialtech.com

Re: Pg+Linux swap use

Andrew Sullivan <andrew@libertyrms.info> — 2003-10-31T17:06:37Z

On Fri, Oct 31, 2003 at 10:42:06AM -0600, Rob Sell wrote:
> For the record I am running on SuSE with a pretty much stock kernel. Not to
> sound na?ve, but is turning of HT something done in the bios?

As far as I know, yes.

A

-- 
----
Andrew Sullivan                         204-4141 Yonge Street
Afilias Canada                        Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110

Re: Pg+Linux swap use

William Yu <wyu@talisys.com> — 2003-10-31T17:17:26Z

Rob Sell wrote:
> Not being one to hijack threads, but I haven't heard of this performance hit
> when using HT, I have what should all rights be a pretty fast server, dual
> 2.4 Xeons with HT 205gb raid 5 array, 1 gig of memory. And it is only 50% as
> fast as my old server which was a dual AMD MP 1400's with a 45gb raid 5
> array and 1gb of ram. I have read everything I could find on Pg performance
> tweaked all the variables that were suggested and nothing. Which is why I
> subscribed to this list, just been lurking so far but this caught my eye. 

Not to get into a big Intel vs AMD argument but 50% sounds about right. 
Let's first assume that the QS rating for the MP1400 is relatively 
accurate and convert that to a 1.4GHz Xeon. 2.4/1.4 = +71%. Since 
processor performance does not increase linearly with clockspeed, 50% is 
in line with expectations. Then you throw in the fact that (1) QS 
ratings for slower AMD chips are understated (but overstated for the 
fastest chips), (2) AMD uses a point-to-point CPU/memory interface (much 
better for SMP) versus the P4/Xeon's shared bus, (3) Athlon architecture 
is more suited for DB work compared to the P4, I'd say you're lucky to 
see 50% more performance from a Xeon 2.4.

As for HT, I've seen quite a few benchmarks where HT hurts performance. 
The problem is it's not only app and workload specific but also system 
and usage specific. As it involves the internal rescheduling of 
processes, adding more simultaneous processes could help to a point and 
then start hurting or vice-versa.

Re: Pg+Linux swap use

Neil Conway <neilc@samurai.com> — 2003-10-31T18:45:48Z

On Fri, 2003-10-31 at 11:37, Greg Stark wrote:
> My understanding is that the case where HT hurts is precisely your case. When
> you have two real processors with HT the kernel will sometimes schedule two
> jobs on the two virtual processors on the same real processor leaving the two
> virtual processors on the other real processor idle.

If you're seeing this behavior, it's sounds like a bug/deficiency in
your kernel's scheduler: if it is HT-aware, it should go to some lengths
to avoid this kind of processor allocation.

-Neil

Re: Pg+Linux swap use

scott.marlowe <scott.marlowe@ihs.com> — 2003-10-31T18:52:35Z

We had a problem at work that when a windows box would connect to a samba 
share with a lot of files in it, the kswapd was going nuts, even though we 
weren't low on memory at all.  Updating to the 2.4.18 or so of the later 
redhats fixed that issue.  It might be related.  I think the kflush daemon 
can get fooled into thinking it needs to get hoppin right now in older 
2.4.x kernels.

On Fri, 31 Oct 2003, alexandre :: aldeia digital wrote:

> Scott, Jeff and Shridhar:
> 
> 1 GB RAM :)
> 
> The stock kernels are not the same, HyperThreading enabled. 80
> simultaneous connections. sort_mem = 4096
> 
> I will compile my own kernel on this weekend, and I will report
> to the list after.
> 
> Thank's all
> 
> Alexandre
> 
> 
> > Also are two kernels exactly same? In my experience linux kernel behaves
> > slightly different from version to version w.r.t swap aggressiveness...
> >
> >   Shridhar
> >
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 4: Don't 'kill -9' the postmaster
> >
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
> 
>                http://www.postgresql.org/docs/faqs/FAQ.html
> 
>

Re: Pg+Linux swap use

Greg Stark <gsstark@mit.edu> — 2003-11-02T16:33:03Z

William Yu <wyu@talisys.com> writes:

> Rob Sell wrote:
>
> > Not being one to hijack threads, but I haven't heard of this performance hit
> > when using HT, I have what should all rights be a pretty fast server, dual
> > 2.4 Xeons with HT 205gb raid 5 array, 1 gig of memory. And it is only 50% as
> > fast as my old server which was a dual AMD MP 1400's with a 45gb raid 5
> > array and 1gb of ram. 
> 
> Not to get into a big Intel vs AMD argument but 50% sounds about right. Let's
> first assume that the QS rating for the MP1400 is relatively accurate and
> convert that to a 1.4GHz Xeon. 2.4/1.4 = +71%. Since processor performance
> does not increase linearly with clockspeed, 50% is in line with expectations.

Hm. You've read "50% as fast" as "50% faster". 
I wonder which the original poster intended.


-- 
greg

Re: Pg+Linux swap use

Bruce Momjian <pgman@candle.pha.pa.us> — 2003-11-06T22:58:52Z

Greg Stark wrote:
> 
> William Yu <wyu@talisys.com> writes:
> 
> > Rob Sell wrote:
> >
> > > Not being one to hijack threads, but I haven't heard of this performance hit
> > > when using HT, I have what should all rights be a pretty fast server, dual
> > > 2.4 Xeons with HT 205gb raid 5 array, 1 gig of memory. And it is only 50% as
> > > fast as my old server which was a dual AMD MP 1400's with a 45gb raid 5
> > > array and 1gb of ram. 
> > 
> > Not to get into a big Intel vs AMD argument but 50% sounds about right. Let's
> > first assume that the QS rating for the MP1400 is relatively accurate and
> > convert that to a 1.4GHz Xeon. 2.4/1.4 = +71%. Since processor performance
> > does not increase linearly with clockspeed, 50% is in line with expectations.
> 
> Hm. You've read "50% as fast" as "50% faster". 
> I wonder which the original poster intended.

Hyper-threading makes 2 cpus be 4 cpu's, but the 4 cpu's are each only
70% as fast, so HT is taking 2x cpus and making it 4x0.70 cpu's, which
gives 2.80 cpu's, and you get that only if you are hammering all four
cpu's with a full load.  Imagine ifd get two cpu-bound processes on the
first die (first 2 cpu's of 4) and the other CPU die is idle, and you
can see that HT isn't all that useful unless you are sure to keep all 4
cpu's busy.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073