Thread

Re: Wierd context-switching issue on Xeon

Anjan Dave <adave@vantage.com> — 2004-04-20T16:48:58Z

If this helps - 

Quad 2.0GHz XEON with highest load we have seen on the applications, DB performing great - 

   procs                      memory      swap          io     system      cpu
 r  b  w   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id
 1  0  0   1616 351820  66144 10813704    0    0     2     0    1     1  0  2  7
 3  0  0   1616 349712  66144 10813736    0    0     8  1634 1362  4650  4  2 95
 0  0  0   1616 347768  66144 10814120    0    0   188  1218 1158  4203  5  1 93
 0  0  1   1616 346596  66164 10814184    0    0     8  1972 1394  4773  4  1 94
 2  0  1   1616 345424  66164 10814272    0    0    20  1392 1184  4197  4  2 94

Around 4k CS/sec
Chipset is Intel ServerWorks GC-HE.
Linux Kernel 2.4.20-28.9bigmem #1 SMP

Thanks,
Anjan


-----Original Message-----
From: Dirk Lutzebäck [mailto:lutzeb@aeccom.com] 
Sent: Tuesday, April 20, 2004 10:29 AM
To: Tom Lane; Josh Berkus
Cc: pgsql-performance@postgreSQL.org; Neil Conway
Subject: Re: [PERFORM] Wierd context-switching issue on Xeon

Dirk Lutzebaeck wrote:

> c) Dual XEON DP, non-bigmem, HT on, E7500 Intel chipset (Supermicro)
>
> performs well and I could not observe context switch peaks here (one 
> user active), almost no extra semop calls

Did Tom's test here: with 2 processes I'll reach 200k+ CS with peaks to 
300k CS. Bummer.. Josh, I don't think you can bash the ServerWorks 
chipset here nor bigmem.

Dirk



---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

               http://archives.postgresql.org

Re: Wierd context-switching issue on Xeon

Josh Berkus <josh@agliodbs.com> — 2004-04-20T16:59:52Z

Anjan,

> Quad 2.0GHz XEON with highest load we have seen on the applications, DB
> performing great -

Can you run Tom's test?   It takes a particular pattern of data access to 
reproduce the issue.

-- 
Josh Berkus
Aglio Database Solutions
San Francisco

Re: Wierd context-switching issue on Xeon

Dave Cramer <pg@fastcrypt.com> — 2004-04-21T02:41:03Z

I modified the code in s_lock.c to remove the spins

#define SPINS_PER_DELAY         1

and it doesn't exhibit the behaviour

This effectively changes the code to 


while(TAS(lock))
	select(10000); // 10ms

Can anyone explain why executing TAS 100 times would increase context
switches ?

Dave


On Tue, 2004-04-20 at 12:59, Josh Berkus wrote:
> Anjan,
> 
> > Quad 2.0GHz XEON with highest load we have seen on the applications, DB
> > performing great -
> 
> Can you run Tom's test?   It takes a particular pattern of data access to 
> reproduce the issue.
-- 
Dave Cramer
519 939 0336
ICQ # 14675561

Re: Wierd context-switching issue on Xeon

Paul Tuckfield <paul@tuckfield.com> — 2004-04-21T18:19:46Z

Dave:

Why would test and set increase context swtches:
Note that it *does not increase* context swtiches when the two threads 
are on the two cores of a single Xeon processor. (use taskset to force 
affinity on linux)

Scenario:
If the two test and set processes are testing and setting the same bit 
as each other, then they'll see worst case cache coherency misses.  
They'll ping a cache line back and forth between CPUs.  Another case, 
might be that they're tesing and setting different bits or words, but 
those bits or words are always in the same cache line, again causing 
worst case cache coherency and misses.  The fact that tis doesn't 
happen when the threads are bound to the 2 cores of a single Xeon 
suggests it's because they're now sharing L1 cache. No pings/bounces.

I wonder do the threads stall so badly when pinging cache lines back 
and forth,  that the kernel sees it as an opportunity to put the 
process to sleep? or do these worst case misses cause an interrupt?

My question is:  What is it that the two threads waiting for when they 
spin? Is it exactly the same resource, or two resources that happen to 
have test-and-set flags in the same cache line?

On Apr 20, 2004, at 7:41 PM, Dave Cramer wrote:

> I modified the code in s_lock.c to remove the spins
>
> #define SPINS_PER_DELAY         1
>
> and it doesn't exhibit the behaviour
>
> This effectively changes the code to
>
>
> while(TAS(lock))
> 	select(10000); // 10ms
>
> Can anyone explain why executing TAS 100 times would increase context
> switches ?
>
> Dave
>
>
> On Tue, 2004-04-20 at 12:59, Josh Berkus wrote:
>> Anjan,
>>
>>> Quad 2.0GHz XEON with highest load we have seen on the applications, 
>>> DB
>>> performing great -
>>
>> Can you run Tom's test?   It takes a particular pattern of data 
>> access to
>> reproduce the issue.
> -- 
> Dave Cramer
> 519 939 0336
> ICQ # 14675561
>
>
> ---------------------------(end of 
> broadcast)---------------------------
> TIP 8: explain analyze is your friend
>

Re: Wierd context-switching issue on Xeon

Tom Lane <tgl@sss.pgh.pa.us> — 2004-04-21T18:51:31Z

Paul Tuckfield <paul@tuckfield.com> writes:
> I wonder do the threads stall so badly when pinging cache lines back 
> and forth,  that the kernel sees it as an opportunity to put the 
> process to sleep? or do these worst case misses cause an interrupt?

No; AFAICS the kernel could not even be aware of that behavior.

The context swap storm is happening because of contention at the next
level up (LWLocks rather than spinlocks).  It could be an independent
issue that just happens to be triggered by the same sort of access
pattern.  I put forward a hypothesis that the cache miss storm caused by
the test-and-set ops induces the context swap storm by making the code
more likely to be executing in certain places at certain times ... but
it's only a hypothesis.

Yesterday evening I had pretty well convinced myself that they were
indeed independent issues: profiling on a single-CPU machine was telling
me that the test case I proposed spends over 10% of its time inside
ReadBuffer, which certainly seems like enough to explain a high rate of
contention on the BufMgrLock, without any assumptions about funny
behavior at the hardware level.  However, your report and Dave's suggest
that there really is some linkage.  So I'm still confused.

			regards, tom lane

Re: Wierd context-switching issue on Xeon

Dave Cramer <pg@fastcrypt.com> — 2004-04-21T19:13:28Z

FYI,

I am doing my testing on non hyperthreading dual athlons. 

Also, the test and set is attempting to set the same resource, and not
simply a bit. It's really an lock;xchg in assemblelr.

Also we are using the PAUSE mnemonic, so we should not be seeing any
cache coherency issues, as the cache is being taken out of the picture
AFAICS ?

Dave

On Wed, 2004-04-21 at 14:19, Paul Tuckfield wrote:
> Dave:
> 
> Why would test and set increase context swtches:
> Note that it *does not increase* context swtiches when the two threads 
> are on the two cores of a single Xeon processor. (use taskset to force 
> affinity on linux)
> 
> Scenario:
> If the two test and set processes are testing and setting the same bit 
> as each other, then they'll see worst case cache coherency misses.  
> They'll ping a cache line back and forth between CPUs.  Another case, 
> might be that they're tesing and setting different bits or words, but 
> those bits or words are always in the same cache line, again causing 
> worst case cache coherency and misses.  The fact that tis doesn't 
> happen when the threads are bound to the 2 cores of a single Xeon 
> suggests it's because they're now sharing L1 cache. No pings/bounces.
> 
> 
> I wonder do the threads stall so badly when pinging cache lines back 
> and forth,  that the kernel sees it as an opportunity to put the 
> process to sleep? or do these worst case misses cause an interrupt?
> 
> My question is:  What is it that the two threads waiting for when they 
> spin? Is it exactly the same resource, or two resources that happen to 
> have test-and-set flags in the same cache line?
> 
> On Apr 20, 2004, at 7:41 PM, Dave Cramer wrote:
> 
> > I modified the code in s_lock.c to remove the spins
> >
> > #define SPINS_PER_DELAY         1
> >
> > and it doesn't exhibit the behaviour
> >
> > This effectively changes the code to
> >
> >
> > while(TAS(lock))
> > 	select(10000); // 10ms
> >
> > Can anyone explain why executing TAS 100 times would increase context
> > switches ?
> >
> > Dave
> >
> >
> > On Tue, 2004-04-20 at 12:59, Josh Berkus wrote:
> >> Anjan,
> >>
> >>> Quad 2.0GHz XEON with highest load we have seen on the applications, 
> >>> DB
> >>> performing great -
> >>
> >> Can you run Tom's test?   It takes a particular pattern of data 
> >> access to
> >> reproduce the issue.
> > -- 
> > Dave Cramer
> > 519 939 0336
> > ICQ # 14675561
> >
> >
> > ---------------------------(end of 
> > broadcast)---------------------------
> > TIP 8: explain analyze is your friend
> >
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 9: the planner will ignore your desire to choose an index scan if your
>       joining column's datatypes do not match
> 
> 
> 
> !DSPAM:4086c4d0263544680737483!
> 
> 
-- 
Dave Cramer
519 939 0336
ICQ # 14675561

Re: Wierd context-switching issue on Xeon patch for 7.4.1

Dave Cramer <pg@fastcrypt.com> — 2004-04-21T20:49:48Z
```
attached.
-- 
Dave Cramer
519 939 0336
ICQ # 14675561
```

Re: Wierd context-switching issue on Xeon

Kenneth Marshall <ktm@it.is.rice.edu> — 2004-04-21T22:02:31Z

On Wed, Apr 21, 2004 at 02:51:31PM -0400, Tom Lane wrote:
> The context swap storm is happening because of contention at the next
> level up (LWLocks rather than spinlocks).  It could be an independent
> issue that just happens to be triggered by the same sort of access
> pattern.  I put forward a hypothesis that the cache miss storm caused by
> the test-and-set ops induces the context swap storm by making the code
> more likely to be executing in certain places at certain times ... but
> it's only a hypothesis.
> 
If the context swap storm derives from LWLock contention, maybe using
a random order to assign buffer locks in buf_init.c would prevent
simple adjacency of buffer allocation to cause the storm. Just offsetting
the assignment by the cacheline size should work. I notice that when
initializing the buffers in shared memory, both the buf->meta_data_lock
and the buf->cntx_lock are immediately adjacent in memory. I am not
familiar enough with the flow through postgres to see if there could
be "fighting" for those two locks. If so, offsetting those by the cache
line size would also stop the context swap storm.

--Ken

Re: Wierd context-switching issue on Xeon

Tom Lane <tgl@sss.pgh.pa.us> — 2004-04-22T01:45:54Z

Kenneth Marshall <ktm@is.rice.edu> writes:
> If the context swap storm derives from LWLock contention, maybe using
> a random order to assign buffer locks in buf_init.c would prevent
> simple adjacency of buffer allocation to cause the storm.

Good try, but no cigar ;-).  The test cases I've been looking at take
only shared locks on the per-buffer locks, so that's not where the
context swaps are coming from.  The swaps have to be caused by the
BufMgrLock, because that's the only exclusive lock being taken.

I did try increasing the allocated size of the spinlocks to 128 bytes
to see if it would do anything.  It didn't ...

			regards, tom lane

Re: Wierd context-switching issue on Xeon patch for 7.4.1

Tom Lane <tgl@sss.pgh.pa.us> — 2004-04-22T02:35:30Z

Dave Cramer <pg@fastcrypt.com> writes:
> diff -c -r1.16 s_lock.c
> *** backend/storage/lmgr/s_lock.c	8 Aug 2003 21:42:00 -0000	1.16
> --- backend/storage/lmgr/s_lock.c	21 Apr 2004 20:27:34 -0000
> ***************
> *** 76,82 ****
>   	 * The select() delays are measured in centiseconds (0.01 sec) because 10
>   	 * msec is a common resolution limit at the OS level.
>   	 */
> ! #define SPINS_PER_DELAY		100
>   #define NUM_DELAYS			1000
>   #define MIN_DELAY_CSEC		1
>   #define MAX_DELAY_CSEC		100
> --- 76,82 ----
>   	 * The select() delays are measured in centiseconds (0.01 sec) because 10
>   	 * msec is a common resolution limit at the OS level.
>   	 */
> ! #define SPINS_PER_DELAY		10
>   #define NUM_DELAYS			1000
>   #define MIN_DELAY_CSEC		1
>   #define MAX_DELAY_CSEC		100

As far as I can tell, this does reduce the rate of semop's
significantly, but it does so by bringing the overall processing rate
to a crawl :-(.  I see 97% CPU idle time when using this patch.
I believe what is happening is that the select() delay in s_lock.c is
being hit frequently because the spin loop isn't allowed to run long
enough to let the other processor get out of the spinlock.

			regards, tom lane

Re: Wierd context-switching issue on Xeon patch for 7.4.1

Josh Berkus <josh@agliodbs.com> — 2004-04-22T02:53:24Z

Tom,

> As far as I can tell, this does reduce the rate of semop's
> significantly, but it does so by bringing the overall processing rate
> to a crawl :-(.  I see 97% CPU idle time when using this patch.
> I believe what is happening is that the select() delay in s_lock.c is
> being hit frequently because the spin loop isn't allowed to run long
> enough to let the other processor get out of the spinlock.

Also, I tested it on production data, and it reduces the CSes by about 40%.  
An improvement, but not a magic bullet.

-- 
Josh Berkus
Aglio Database Solutions
San Francisco

Re: Wierd context-switching issue on Xeon patch for 7.4.1

Dave Cramer <pg@fastcrypt.com> — 2004-04-22T03:06:41Z

Yeah, I did some more testing myself, and actually get better numbers
with increasing spins per delay to 1000, but my suspicion is that it is
highly dependent on finding the right delay for the processor you are
on.

My hypothesis is that if you spin approximately the same or more time
than the average time it takes to get finished with the shared resource
then this should reduce cs.

Certainly more ideas are required here.

Dave 
On Wed, 2004-04-21 at 22:35, Tom Lane wrote:
> Dave Cramer <pg@fastcrypt.com> writes:
> > diff -c -r1.16 s_lock.c
> > *** backend/storage/lmgr/s_lock.c	8 Aug 2003 21:42:00 -0000	1.16
> > --- backend/storage/lmgr/s_lock.c	21 Apr 2004 20:27:34 -0000
> > ***************
> > *** 76,82 ****
> >   	 * The select() delays are measured in centiseconds (0.01 sec) because 10
> >   	 * msec is a common resolution limit at the OS level.
> >   	 */
> > ! #define SPINS_PER_DELAY		100
> >   #define NUM_DELAYS			1000
> >   #define MIN_DELAY_CSEC		1
> >   #define MAX_DELAY_CSEC		100
> > --- 76,82 ----
> >   	 * The select() delays are measured in centiseconds (0.01 sec) because 10
> >   	 * msec is a common resolution limit at the OS level.
> >   	 */
> > ! #define SPINS_PER_DELAY		10
> >   #define NUM_DELAYS			1000
> >   #define MIN_DELAY_CSEC		1
> >   #define MAX_DELAY_CSEC		100
> 
> 
> As far as I can tell, this does reduce the rate of semop's
> significantly, but it does so by bringing the overall processing rate
> to a crawl :-(.  I see 97% CPU idle time when using this patch.
> I believe what is happening is that the select() delay in s_lock.c is
> being hit frequently because the spin loop isn't allowed to run long
> enough to let the other processor get out of the spinlock.
> 
> 			regards, tom lane
> 
> 
> 
> !DSPAM:40872f7e21492906114513!
> 
> 
-- 
Dave Cramer
519 939 0336
ICQ # 14675561

Re: Wierd context-switching issue on Xeon patch for 7.4.1

Dave Cramer <pg@fastcrypt.com> — 2004-04-22T03:18:47Z

More data....

On a dual xeon with HTT enabled:

I tried increasing the NUM_SPINS to 1000 and it works better.

NUM_SPINLOCKS	CS	ID	pgbench

100		250K	59%	230 TPS
1000		125K	55%	228 TPS

This is certainly heading in the right direction ? Although it looks
like it is highly dependent on the system you are running on.

--dc--	 



On Wed, 2004-04-21 at 22:53, Josh Berkus wrote:
> Tom,
> 
> > As far as I can tell, this does reduce the rate of semop's
> > significantly, but it does so by bringing the overall processing rate
> > to a crawl :-(.  I see 97% CPU idle time when using this patch.
> > I believe what is happening is that the select() delay in s_lock.c is
> > being hit frequently because the spin loop isn't allowed to run long
> > enough to let the other processor get out of the spinlock.
> 
> Also, I tested it on production data, and it reduces the CSes by about 40%.  
> An improvement, but not a magic bullet.
-- 
Dave Cramer
519 939 0336
ICQ # 14675561

Re: Wierd context-switching issue on Xeon patch for 7.4.1

Tom Lane <tgl@sss.pgh.pa.us> — 2004-04-22T04:23:24Z

Dave Cramer <pg@fastcrypt.com> writes:
> I tried increasing the NUM_SPINS to 1000 and it works better.

Doesn't surprise me.  The value of 100 is about right on the assumption
that the spinlock instruction per se is not too much more expensive than
any other instruction.  What I was seeing from oprofile suggested that
the spinlock instruction cost about 100x more than an ordinary
instruction :-( ... so maybe 200 or so would be good on a Xeon.

> This is certainly heading in the right direction ? Although it looks
> like it is highly dependent on the system you are running on.

Yeah.  I don't know a reasonable way to tune this number automatically
for particular systems ... but at the very least we'd need to find a way
to distinguish uniprocessor from multiprocessor, because on a
uniprocessor the optimal value is surely 1.

			regards, tom lane

Re: Wierd context-switching issue on Xeon patch for 7.4.1

Christopher Kings-Lynne <chriskl@familyhealth.com.au> — 2004-04-22T04:44:07Z

> Yeah.  I don't know a reasonable way to tune this number automatically
> for particular systems ... but at the very least we'd need to find a way
> to distinguish uniprocessor from multiprocessor, because on a
> uniprocessor the optimal value is surely 1.

 From TODO:

* Add code to detect an SMP machine and handle spinlocks accordingly 
from distributted.net, http://www1.distributed.net/source, in 
client/common/cpucheck.cpp

Chris

Re: Wierd context-switching issue on Xeon patch for 7.4.1

Bruce Momjian <pgman@candle.pha.pa.us> — 2004-04-22T04:56:15Z

Tom Lane wrote:
> Dave Cramer <pg@fastcrypt.com> writes:
> > I tried increasing the NUM_SPINS to 1000 and it works better.
> 
> Doesn't surprise me.  The value of 100 is about right on the assumption
> that the spinlock instruction per se is not too much more expensive than
> any other instruction.  What I was seeing from oprofile suggested that
> the spinlock instruction cost about 100x more than an ordinary
> instruction :-( ... so maybe 200 or so would be good on a Xeon.
> 
> > This is certainly heading in the right direction ? Although it looks
> > like it is highly dependent on the system you are running on.
> 
> Yeah.  I don't know a reasonable way to tune this number automatically
> for particular systems ... but at the very least we'd need to find a way
> to distinguish uniprocessor from multiprocessor, because on a
> uniprocessor the optimal value is surely 1.

Have you looked at the code pointed to by our TODO item:
	
	* Add code to detect an SMP machine and handle spinlocks accordingly
	  from distributted.net, http://www1.distributed.net/source,
	  in client/common/cpucheck.cpp

For BSDOS it has:

    #if (CLIENT_OS == OS_FREEBSD) || (CLIENT_OS == OS_BSDOS) || \
        (CLIENT_OS == OS_OPENBSD) || (CLIENT_OS == OS_NETBSD)
    { /* comment out if inappropriate for your *bsd - cyp (25/may/1999) */
      int ncpus; size_t len = sizeof(ncpus);
      int mib[2]; mib[0] = CTL_HW; mib[1] = HW_NCPU;
      if (sysctl( &mib[0], 2, &ncpus, &len, NULL, 0 ) == 0)
      //if (sysctlbyname("hw.ncpu", &ncpus, &len, NULL, 0 ) == 0)
        cpucount = ncpus;
    }

and I can confirm that on my computer it works:

	hw.ncpu = 2

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: Wierd context-switching issue on Xeon patch for 7.4.1

Tom Lane <tgl@sss.pgh.pa.us> — 2004-04-22T05:13:47Z

Bruce Momjian <pgman@candle.pha.pa.us> writes:
> For BSDOS it has:

>     #if (CLIENT_OS == OS_FREEBSD) || (CLIENT_OS == OS_BSDOS) || \
>         (CLIENT_OS == OS_OPENBSD) || (CLIENT_OS == OS_NETBSD)
>     { /* comment out if inappropriate for your *bsd - cyp (25/may/1999) */
>       int ncpus; size_t len = sizeof(ncpus);
>       int mib[2]; mib[0] = CTL_HW; mib[1] = HW_NCPU;
>       if (sysctl( &mib[0], 2, &ncpus, &len, NULL, 0 ) == 0)
>       //if (sysctlbyname("hw.ncpu", &ncpus, &len, NULL, 0 ) == 0)
>         cpucount = ncpus;
>     }

Multiplied by how many platforms?  Ewww...

I was wondering about some sort of dynamic adaptation, roughly along the
lines of "whenever a spin loop successfully gets the lock after
spinning, decrease the allowed loop count by one; whenever we fail to
get the lock after spinning, increase by 100; if the loop count reaches,
say, 10000, decide we are on a uniprocessor and irreversibly set it to
1."  As written this would tend to incur a select() delay once per
hundred spinlock acquisitions, which is way too much, but I think we
could make it work with a sufficiently slow adaptation rate.  The tricky
part is that a slow adaptation rate means we can't have every backend
figuring this out for itself --- the right value would have to be
maintained globally, and I'm not sure how to do that without adding a
lot of overhead.

			regards, tom lane

Re: Wierd context-switching issue on Xeon patch for 7.4.1

Tom Lane <tgl@sss.pgh.pa.us> — 2004-04-22T12:36:28Z

Dave Cramer <pg@fastcrypt.com> writes:
> My hypothesis is that if you spin approximately the same or more time
> than the average time it takes to get finished with the shared resource
> then this should reduce cs.

The only thing we use spinlocks for nowadays is to protect LWLocks, so
the "average time" involved is fairly small and stable --- or at least
that was the design intention.  What we seem to be seeing is that on SMP
machines, cache coherency issues cause the TAS step itself to be
expensive and variable.  However, in the experiments I did, strace'ing
showed that actual spin timeouts (manifested by the execution of a
delaying select()) weren't actually that common; the big source of
context switches is semop(), which indicates contention at the LWLock
level rather than the spinlock level.  So while tuning the spinlock
limit count might be a useful thing to do in general, I think it will
have only negligible impact on the particular problems we're discussing
in this thread.

			regards, tom lane

Re: Wierd context-switching issue on Xeon patch for 7.4.1

Josh Berkus <josh@agliodbs.com> — 2004-04-22T17:37:10Z

Tom,

> The tricky
> part is that a slow adaptation rate means we can't have every backend
> figuring this out for itself --- the right value would have to be
> maintained globally, and I'm not sure how to do that without adding a
> lot of overhead.

This may be a moot point, since you've stated that changing the loop timing 
won't solve the problem, but what about making the test part of make?   I 
don't think too many systems are going to change processor architectures once 
in production, and those that do can be told to re-compile.

-- 
Josh Berkus
Aglio Database Solutions
San Francisco

Re: Wierd context-switching issue on Xeon patch for 7.4.1

Tom Lane <tgl@sss.pgh.pa.us> — 2004-04-22T17:55:53Z

Josh Berkus <josh@agliodbs.com> writes:
> This may be a moot point, since you've stated that changing the loop timing 
> won't solve the problem, but what about making the test part of make?   I 
> don't think too many systems are going to change processor architectures once
> in production, and those that do can be told to re-compile.

Having to recompile to run on single- vs dual-processor machines doesn't
seem like it would fly.

			regards, tom lane

Re: Wierd context-switching issue on Xeon patch for 7.4.1

Josh Berkus <josh@agliodbs.com> — 2004-04-22T18:11:43Z

Tom,

> Having to recompile to run on single- vs dual-processor machines doesn't
> seem like it would fly.

Oh, I don't know.  Many applications require compiling for a target 
architecture; SQL Server, for example, won't use a 2nd processor without 
re-installation.   I'm not sure about Oracle.

It certainly wasn't too long ago that Linux gurus were esposing re-compiling 
the kernel for the machine.

And it's not like they would *have* to re-compile to use PostgreSQL after 
adding an additional processor.  Just if they wanted to maximize peformance 
benefit.

Also, this is a fairly rare circumstance, I think; to judge by my clients, 
once a database server is in production nobody touches the hardware.

-- 
-Josh Berkus
 Aglio Database Solutions
 San Francisco

Re: Wierd context-switching issue on Xeon patch for 7.4.1

Bruce Momjian <pgman@candle.pha.pa.us> — 2004-04-22T18:31:22Z

Josh Berkus wrote:
> Tom,
> 
> > Having to recompile to run on single- vs dual-processor machines doesn't
> > seem like it would fly.
> 
> Oh, I don't know.  Many applications require compiling for a target 
> architecture; SQL Server, for example, won't use a 2nd processor without 
> re-installation.   I'm not sure about Oracle.
> 
> It certainly wasn't too long ago that Linux gurus were esposing re-compiling 
> the kernel for the machine.
> 
> And it's not like they would *have* to re-compile to use PostgreSQL after 
> adding an additional processor.  Just if they wanted to maximize peformance 
> benefit.
> 
> Also, this is a fairly rare circumstance, I think; to judge by my clients, 
> once a database server is in production nobody touches the hardware.

A much simpler solution would be for the postmaster to run a test during
startup.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: Wierd context-switching issue on Xeon patch for 7.4.1

Rod Taylor <pg@rbt.ca> — 2004-04-22T21:22:11Z

On Thu, 2004-04-22 at 13:55, Tom Lane wrote:
> Josh Berkus <josh@agliodbs.com> writes:
> > This may be a moot point, since you've stated that changing the loop timing 
> > won't solve the problem, but what about making the test part of make?   I 
> > don't think too many systems are going to change processor architectures once
> > in production, and those that do can be told to re-compile.
> 
> Having to recompile to run on single- vs dual-processor machines doesn't
> seem like it would fly.

Is it something the postmaster could quickly determine and set a global
during the startup cycle?

Re: Wierd context-switching issue on Xeon patch for 7.4.1

Andrew McMillan <andrew@catalyst.net.nz> — 2004-04-25T07:13:35Z

On Thu, 2004-04-22 at 10:37 -0700, Josh Berkus wrote:
> Tom,
> 
> > The tricky
> > part is that a slow adaptation rate means we can't have every backend
> > figuring this out for itself --- the right value would have to be
> > maintained globally, and I'm not sure how to do that without adding a
> > lot of overhead.
> 
> This may be a moot point, since you've stated that changing the loop timing 
> won't solve the problem, but what about making the test part of make?   I 
> don't think too many systems are going to change processor architectures once 
> in production, and those that do can be told to re-compile.

Sure they do - PostgreSQL is regularly provided as a pre-compiled
distribution.  I haven't compiled PostgreSQL for years, and we have it
running on dozens of machines, some SMP, some not, but most running
Debian Linux.

Even having a compiler _installed_ on one of our client's database
servers would usually be considered against security procedures, and
would get a black mark when the auditors came through.

Regards,
					Andrew McMillan
-------------------------------------------------------------------------
Andrew @ Catalyst .Net .NZ  Ltd,  PO Box 11-053,  Manners St,  Wellington
WEB: http://catalyst.net.nz/             PHYS: Level 2, 150-154 Willis St
DDI: +64(4)916-7201       MOB: +64(21)635-694      OFFICE: +64(4)499-2267
                     Planning an election?  Call us!
-------------------------------------------------------------------------

Re: Wierd context-switching issue on Xeon patch for 7.4.1

Josh Berkus <josh@agliodbs.com> — 2004-04-27T00:03:25Z

Dave,

> Yeah, I did some more testing myself, and actually get better numbers
> with increasing spins per delay to 1000, but my suspicion is that it is
> highly dependent on finding the right delay for the processor you are
> on.

Well, it certainly didn't help here:

procs                      memory      swap          io     system         cpu
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  0      0 14870744 123872 1129912    0    0     0     0 1027 187341 48 27 
26  0
 2  0      0 14869912 123872 1129912    0    0     0    48 1030 126490 65 18 
16  0
 2  0      0 14867032 123872 1129912    0    0     0     0 1021 106046 72 16 
12  0
 2  0      0 14869912 123872 1129912    0    0     0     0 1025 90256 76 14 10  
0
 2  0      0 14870424 123872 1129912    0    0     0     0 1022 135249 63 22 
16  0
 2  0      0 14872664 123872 1129912    0    0     0     0 1023 131111 63 20 
17  0
 1  0      0 14871128 123872 1129912    0    0     0    48 1024 155728 57 22 
20  0
 2  0      0 14871128 123872 1129912    0    0     0     0 1028 189655 49 29 
22  0
 2  0      0 14871064 123872 1129912    0    0     0     0 1018 190744 48 29 
23  0
 2  0      0 14871064 123872 1129912    0    0     0     0 1027 186812 51 26 
23  0


-- 
-Josh Berkus
 Aglio Database Solutions
 San Francisco

Re: Wierd context-switching issue on Xeon patch for 7.4.1

Dave Cramer <pg@fastcrypt.com> — 2004-04-27T00:16:00Z

Are you testing this with Tom's code, you need to do a baseline
measurement with 10 and then increase it, you will still get lots of cs,
but it will be less.

Dave
On Mon, 2004-04-26 at 20:03, Josh Berkus wrote:
> Dave,
> 
> > Yeah, I did some more testing myself, and actually get better numbers
> > with increasing spins per delay to 1000, but my suspicion is that it is
> > highly dependent on finding the right delay for the processor you are
> > on.
> 
> Well, it certainly didn't help here:
> 
> procs                      memory      swap          io     system         cpu
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
>  2  0      0 14870744 123872 1129912    0    0     0     0 1027 187341 48 27 
> 26  0
>  2  0      0 14869912 123872 1129912    0    0     0    48 1030 126490 65 18 
> 16  0
>  2  0      0 14867032 123872 1129912    0    0     0     0 1021 106046 72 16 
> 12  0
>  2  0      0 14869912 123872 1129912    0    0     0     0 1025 90256 76 14 10  
> 0
>  2  0      0 14870424 123872 1129912    0    0     0     0 1022 135249 63 22 
> 16  0
>  2  0      0 14872664 123872 1129912    0    0     0     0 1023 131111 63 20 
> 17  0
>  1  0      0 14871128 123872 1129912    0    0     0    48 1024 155728 57 22 
> 20  0
>  2  0      0 14871128 123872 1129912    0    0     0     0 1028 189655 49 29 
> 22  0
>  2  0      0 14871064 123872 1129912    0    0     0     0 1018 190744 48 29 
> 23  0
>  2  0      0 14871064 123872 1129912    0    0     0     0 1027 186812 51 26 
> 23  0
-- 
Dave Cramer
519 939 0336
ICQ # 14675561

Re: Wierd context-switching issue on Xeon patch for 7.4.1

Josh Berkus <josh@agliodbs.com> — 2004-04-27T18:05:00Z

Dave,

> Are you testing this with Tom's code, you need to do a baseline
> measurement with 10 and then increase it, you will still get lots of cs,
> but it will be less.

No, that was just a test of 1000 straight up.    Tom outlined a method, but I 
didn't see any code that would help me find a better level, other than just 
trying each +100 increase one at a time.   This would take days of testing 
...
-- 
Josh Berkus
Aglio Database Solutions
San Francisco

Re: Wierd context-switching issue on Xeon patch for 7.4.1

Dave Cramer <pg@fastcrypt.com> — 2004-04-27T18:27:33Z

Josh,

I think you can safely increase by orders of magnitude here, instead of
by +100, my wild ass guess is that the sweet spot is the spin time
should be approximately the time it takes to consume the resource. So if
you have a really fast machine then the spin count should be higher. 

Also you have to take into consideration your memory bus speed, with the
pause instruction inserted in the loop the timing is now dependent on
memory speed.

But... you need a baseline first.

Dave
On Tue, 2004-04-27 at 14:05, Josh Berkus wrote:
> Dave,
> 
> > Are you testing this with Tom's code, you need to do a baseline
> > measurement with 10 and then increase it, you will still get lots of cs,
> > but it will be less.
> 
> No, that was just a test of 1000 straight up.    Tom outlined a method, but I 
> didn't see any code that would help me find a better level, other than just 
> trying each +100 increase one at a time.   This would take days of testing 
> ...
-- 
Dave Cramer
519 939 0336
ICQ # 14675561

Re: Wierd context-switching issue on Xeon patch for 7.4.1

Josh Berkus <josh@agliodbs.com> — 2004-04-27T21:03:13Z

Dave,

> But... you need a baseline first.

A baseline on CS?   I have that ....

-- 
-Josh Berkus
 Aglio Database Solutions
 San Francisco