Horton

Thread

  1. Re: [HACKERS] Everything leaks; How it mm suppose to work?

    Maurice Gittens <mgittens@gits.nl> — 1998-04-02T16:02:25Z

    >
>Running postgresql in interactive mode shows that for each query I 
>type there is memory lost. The exact amount of memory lost depends on
>the query I use. The amount of memory not freed is also a function
>of the number of tuples returned.
>
 
Oops, it seems some palloced memory is not freed by pfree but
using some other function(s).
My mistake, sorry.

Thanks, with regards from Maurice.

  2. Re: [HACKERS] Everything leaks; How it mm suppose to work?

    Bruce Momjian <maillist@candle.pha.pa.us> — 1998-04-04T21:07:12Z

    > 
> 
> >
> >Running postgresql in interactive mode shows that for each query I 
> >type there is memory lost. The exact amount of memory lost depends on
> >the query I use. The amount of memory not freed is also a function
> >of the number of tuples returned.
> >
>  
> Oops, it seems some palloced memory is not freed by pfree but
> using some other function(s).
> My mistake, sorry.
> 

One thing I have found is that:

	select * from test where 1=0;

do not leak memory while

	select * from test where x=-999;

does leak memory, even though neither returns any rows.  Strange.  Would
seem to point to the optimizer or executor.

-- 
Bruce Momjian                          |  830 Blythe Avenue
maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
  +  If your life is a hard drive,     |  (610) 353-9879(w)
  +  Christ can be your backup.        |  (610) 853-3000(h)

  3. Re: [HACKERS] Everything leaks; How it mm suppose to work?

    Bruce Momjian <maillist@candle.pha.pa.us> — 1998-04-04T21:59:09Z

    > 
> 
> >
> >Running postgresql in interactive mode shows that for each query I 
> >type there is memory lost. The exact amount of memory lost depends on
> >the query I use. The amount of memory not freed is also a function
> >of the number of tuples returned.
> >
>  
> Oops, it seems some palloc'ed memory is not freed by pfree() but
> using some other function(s).
> My mistake, sorry.

OK, I think I know where the leak is from.  I checked AllocSetDump(), and
it did not show any problems with any context growing, so I started to
suspect direct calls to malloc().  I tried trapping on malloc(), but
there are too many calls.

Then I ran profiling on the two queries I mentioned, where one leaks and
one doesn't, and found that the leaking one had 500 extra calls to
malloc.  Grep'ing out the calls and comparing the output of the two
profiles, I found:

                0.00        0.00       1/527         ___irs_gen_acc [162]
                0.00        0.00      35/527         _GetDatabasePath [284]
                0.00        0.00     491/527         _GetDatabaseName [170]
[166]    0.8    0.00        0.00     527         _strdup [166]
                0.00        0.00     527/2030        _malloc [105]
                0.00        0.00     527/604         _strlen [508]
                0.00        0.00     527/532         _bcopy [515]


I believe this code was modified by Vadim to fix our problem with blind
write errors when using psql while the regression tests were being run.

Am I correct on this?  I have not developed a fix yet.

-- 
Bruce Momjian                          |  830 Blythe Avenue
maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
  +  If your life is a hard drive,     |  (610) 353-9879(w)
  +  Christ can be your backup.        |  (610) 853-3000(h)

  4. Re: [HACKERS] Everything leaks; How it mm suppose to work?

    Vadim B. Mikheev <vadim@sable.krasnoyarsk.su> — 1998-04-05T13:39:03Z

    Bruce Momjian wrote:
> 
> Then I ran profiling on the two queries I mentioned, where one leaks and
> one doesn't, and found that the leaking one had 500 extra calls to
> malloc.  Grep'ing out the calls and comparing the output of the two
> profiles, I found:
> 
>                 0.00        0.00       1/527         ___irs_gen_acc [162]
>                 0.00        0.00      35/527         _GetDatabasePath [284]
>                 0.00        0.00     491/527         _GetDatabaseName [170]
> [166]    0.8    0.00        0.00     527         _strdup [166]
>                 0.00        0.00     527/2030        _malloc [105]
>                 0.00        0.00     527/604         _strlen [508]
>                 0.00        0.00     527/532         _bcopy [515]
> 
> I believe this code was modified by Vadim to fix our problem with blind
> write errors when using psql while the regression tests were being run.
> 
> Am I correct on this?  I have not developed a fix yet.

Only src/backend/storage/smgr/md.c:mdblindwrt() was changed...

Vadim

  5. Re: [HACKERS] Everything leaks; How it mm suppose to work?

    David Gould <dg@illustra.com> — 1998-04-09T07:48:17Z

    Bruce:
> Maurice:
> > >Running postgresql in interactive mode shows that for each query I 
> > >type there is memory lost. The exact amount of memory lost depends on
> > >the query I use. The amount of memory not freed is also a function
> > >of the number of tuples returned.
> > >
> >  
> > Oops, it seems some palloced memory is not freed by pfree but
> > using some other function(s).
> > My mistake, sorry.
> > 
> 
> One thing I have found is that:
> 
> 	select * from test where 1=0;
> 
> do not leak memory while
> 
> 	select * from test where x=-999;
> 
> does leak memory, even though neither returns any rows.  Strange.  Would
> seem to point to the optimizer or executor.

In the first case, the where clause is constant and evaluates false, so
not much is done. In the second case, the table is scanned and presumably
some memory is allocated for each row, probably to evaluate the expression.
Since this memory is allocated into a per statement duration, it will not
be freed until the end of the statement when the context is destroyed.

-dg


David Gould            dg@illustra.com           510.628.3783 or 510.305.9468 
Informix Software  (No, really)         300 Lakeside Drive  Oakland, CA 94612
 - Linux. Not because it is free. Because it is better.

  6. Re: [HACKERS] Everything leaks; How it mm suppose to work?

    Massimo Dal Zotto <dz@cs.unitn.it> — 1998-04-09T10:31:42Z

    > 
> Bruce:
> > Maurice:
> > > >Running postgresql in interactive mode shows that for each query I 
> > > >type there is memory lost. The exact amount of memory lost depends on
> > > >the query I use. The amount of memory not freed is also a function
> > > >of the number of tuples returned.
> > > >
> > >  
> > > Oops, it seems some palloced memory is not freed by pfree but
> > > using some other function(s).
> > > My mistake, sorry.
> > > 
> > 
> > One thing I have found is that:
> > 
> > 	select * from test where 1=0;
> > 
> > do not leak memory while
> > 
> > 	select * from test where x=-999;
> > 
> > does leak memory, even though neither returns any rows.  Strange.  Would
> > seem to point to the optimizer or executor.
> 
> In the first case, the where clause is constant and evaluates false, so
> not much is done. In the second case, the table is scanned and presumably
> some memory is allocated for each row, probably to evaluate the expression.
> Since this memory is allocated into a per statement duration, it will not
> be freed until the end of the statement when the context is destroyed.
> 
> -dg
> 

Does it make sense to have a 'row' context which is released just before
starting with a new tuple ? The total number or free is the same but they
are distributed over the query and unused memory should not accumulate.
I have seen backends growing to 40-60MB with queries which scan a very
large number of rows.

Massimo Dal Zotto

+----------------------------------------------------------------------+
|  Massimo Dal Zotto                e-mail:  dz@cs.unitn.it            |
|  Via Marconi, 141                 phone:  ++39-461-534251            |
|  38057 Pergine Valsugana (TN)     www:  http://www.cs.unitn.it/~dz/  |
|  Italy                            pgp:  finger dz@tango.cs.unitn.it  |
+----------------------------------------------------------------------+

  7. Re: [HACKERS] Everything leaks; How it mm suppose to work?

    Bruce Momjian <maillist@candle.pha.pa.us> — 1998-04-09T13:51:44Z

    > 
> Bruce:
> > Maurice:
> > > >Running postgresql in interactive mode shows that for each query I 
> > > >type there is memory lost. The exact amount of memory lost depends on
> > > >the query I use. The amount of memory not freed is also a function
> > > >of the number of tuples returned.
> > > >
> > >  
> > > Oops, it seems some palloced memory is not freed by pfree but
> > > using some other function(s).
> > > My mistake, sorry.
> > > 
> > 
> > One thing I have found is that:
> > 
> > 	select * from test where 1=0;
> > 
> > do not leak memory while
> > 
> > 	select * from test where x=-999;
> > 
> > does leak memory, even though neither returns any rows.  Strange.  Would
> > seem to point to the optimizer or executor.
> 
> In the first case, the where clause is constant and evaluates false, so
> not much is done. In the second case, the table is scanned and presumably
> some memory is allocated for each row, probably to evaluate the expression.
> Since this memory is allocated into a per statement duration, it will not
> be freed until the end of the statement when the context is destroyed.

This will be fixed in the 6.3.2 patch, due out soon.

-- 
Bruce Momjian                          |  830 Blythe Avenue
maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
  +  If your life is a hard drive,     |  (610) 353-9879(w)
  +  Christ can be your backup.        |  (610) 853-3000(h)