Noticed something odd with pgbench

Hey guys,

So, we have a pretty beefy system that runs dual X5675's with 72GB of 
RAM. After our recent upgrade to 9.1, things have been... odd. I managed 
to track it down to one setting:

shared_buffers = 8GB

The thing is, we currently have 850 clients connected to our database (I 
know, that's bad, but the platform is not compatible with pgpool or 
pgbouncer right now). So I did a pgbench test (scale = 3000) with 850 
clients. The 3000 scale was enough to cross the NUMA barrier, because 
that should force zone flushing if there's a problem. To cheat a little, 
I preloaded all pgbench tables into memory with dd.

I'm not running pgbench to see performance. In this case, it's a load 
test. And at first, the load test starts normally, and everything looks 
fine. But if I check /proc/meminfo, I see this within 2-5 minutes:

MemFree:          461660 kB
Active(file):   23252240 kB
Inactive(file): 21272440 kB

However, if I change shared_buffers to 4GB, it ends up converging to this:

MemFree:        11024696 kB
Active(file):   46009064 kB
Inactive(file):   239672 kB

If you watch the contents of /proc/meminfo during the pgbench, it's 
pretty clear when the transition starts. It only takes a couple minutes 
of pgbench, and it just spirals out of control, marking tons of file 
cache inactive. But according to the OS, that memory is being paged out 
because the OS is out of genuinely free memory. But how does a 4GB bump 
in shared_buffers wipe out 12GB?

It does the same thing at 6GB. 4GB is safe for hours on end, but 6GB and 
8GB implode within in minutes. During this, kswapd goes crazy paging out 
the cache, at the same time it's reading from disk to put them back in. 
It's like I fed the kernel poison or something.

Has anybody else noticed something like this? I got this behavior with 
9.1.6 on a 3.2 kernel. No amount of tweaks in /proc/sys/vm changed 
anything either, so I'm not convinced it's a NUMA problem.


-- 
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-444-8534
sthomas@optionshouse.com

______________________________________________

See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email