Re: postgresql + apache under heavy load

On Wed, 21 Jan 2004, Alex Madon wrote:

> Hello Scott,
> Thank you for your answer.
> 
> >I'm not familiar with DBX.  Is that connection pooling or what?
> >  
> >
> I could not find this information, sorry.
> 
> >Are you SURE all your memory is in use?  What exactly does top say about 
> >things like cached and buff memory (I'm assuming you're on linux, any 
> >differences in top on another OS would be minor.)  If the kernel still 
> >shows a fair bit of cached and buff memory, your memory is not getting all 
> >used up.
> >
> Well my xosview show that caching begin at a concurrency of 40.
> At 80 my cache begins to be filled completely, so machine having big 
> problems.

I think you're confusing what I meant.  Caching is good.  Swapping is bad.  
Having a large amount of cache is a good thing.  It means the OS is 
caching all your data in memory for faster access.

> >>If I increases to
> >>ab -n 1000 -c 100 http://localsite/testscript
> >>I get this memory problem.
> >>    
> >>
> >
> >Where's the break point?  Just wondering.  Does it show up at 20, 40, 60, 
> >80, or only at 100?  If so, that's really not bad.  
> >
>  Here is some results (I kept -n 100 an just vraied the -c option)
> --c 1
> Failed requests:        0
> Time per request:       322.096 [ms] (mean, across all concurrent requests)
> 
> -c 2
> Failed requests:        0
> Time per request:       374.220 [ms] (mean, across all concurrent requests)
> 
> -c 10
> Failed requests:        68
>  (Connect: 0, Length: 68, Exceptions: 0)
> Time per request:       314.779 [ms] (mean, across all concurrent requests)

OK, there's a problem, you're getting failed requests at -c 10, which 
means you likely have postgresql configured in the wrong 
direction.  configure postgresql to use more memory (sort_mem can be set 
to about 8 megs without a lot of issues on most boxes, going higher may 
use up all your memory in certain situations (high concurrency)).

> For a higher concurrency, the cache is completly filled, and have to 
> reboot the machine.

No, you should NEVER have to reboot a unix box.  period.  filled cache, 
again, is a GOOD THING.   not bad.

> (I didn't leave the system caching forever, just press to reboot 
> button)... could be interesting to wait to see if the systems recovers 
> after a while

Yes, please do.  Also, show us a save of top while under load.

I'm betting your machine has plenty of memory, and is not using it 
effectively, due to postgresql being too conservatively configured.