Re: log chunking broken with large queries under load

On 04/01/2012 06:34 PM, Andrew Dunstan wrote:
> Some of my PostgreSQL Experts colleagues have been complaining to me 
> that servers under load with very large queries cause CSV log files 
> that are corrupted, because lines are apparently multiplexed. The log 
> chunking protocol between the errlog routines and the syslogger is 
> supposed to prevent that, so I did a little work to try to reproduce 
> it in a controlled way.


Well, a little further digging jogged my memory a bit. It looks like we 
underestimated the amount of messages we might get as more than one 
chunk fairly badly.

We could just increase CHUNK_SLOTS in syslogger.c, but I opted instead 
to stripe the slots with a two dimensional array, so we didn't have to 
search a larger number of slots for any given message. See the attached 
patch.

I'm not sure how much we want to scale this up. I set CHUNK_STRIPES to 
20 to start with, and I've asked some colleagues with very heavy log 
loads with very large queries to test it out if possible. If anyone else 
has a similar load I'd appreciate similar testing.

cheers

andrew