Horton
Menu

Re: Amcheck verification of GiST and GIN

Kirill Reshke <reshkekirill@gmail.com>

View thread
From: Kirill Reshke <reshkekirill@gmail.com>
To: Tomas Vondra <tomas@vondra.me>
Cc: "Andrey M. Borodin" <x4mmm@yandex-team.ru>, Tomas Vondra <tomas.vondra@enterprisedb.com>, Alexander Lakhin <exclusion@gmail.com>, Andrey Borodin <amborodin86@gmail.com>, Peter Geoghegan <pg@bowt.ie>, Mark Dilger <mark.dilger@enterprisedb.com>, Jose Arthur Benetasso Villanova <jose.arthur@gmail.com>, Andres Freund <andres@anarazel.de>, Nikolay Samokhvalov <samokhvalov@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2024-10-18T12:42:43Z
Lists: pgsql-hackers

Attachments

Hi!

On Mon, 5 Aug 2024 at 20:05, Tomas Vondra <tomas@vondra.me> wrote:
>
> Hi,
>
> I've spent a bit more time looking at the GiST part as part of my
> "parallel GiST build" patch nearby, and I think there's some sort of
> memory leak.
>
> Consider this:
>
>   create table t (a text);
>
>   insert into t select md5(i::text)
>     from generate_series(1,25000000) s(i);
>
>   create index on t using gist (a gist_trgm_ops);
>
>   select gist_index_check('t_a_idx', true);
>
> This creates a ~4GB GiST trigram index, and then checks it. But that
> gets killed, because of OOM killer. On my test machine it consumes
> ~6.5GB of memory before OOM intervenes.
>
> The memory context stats look like this:
>
>   TopPortalContext: 8192 total in 1 blocks; 7680 free (0 chunks); 512 used
>     PortalContext: 1024 total in 1 blocks; 616 free (0 chunks); 408
> used: <unnamed>
>       ExecutorState: 8192 total in 1 blocks; 4024 free (4 chunks); 4168 used
>         printtup: 8192 total in 1 blocks; 7952 free (0 chunks); 240 used
>         ExprContext: 8192 total in 1 blocks; 7224 free (10 chunks); 968 used
>           amcheck context: 3128950872 total in 376 blocks; 219392 free
> (1044 chunks); 3128731480 used
>             ExecutorState: 8192 total in 1 blocks; 7200 free (0 chunks);
> 992 used
>               ExprContext: 8192 total in 1 blocks; 7952 free (0 chunks);
> 240 used
>             GiST scan context: 22248 total in 2 blocks; 7808 free (8
> chunks); 14440 used
>
> This is from before the OOM kill, but it shows there's ~3GB of memory is
> the amcheck context.
>
> Seems like a memory leak to me - I didn't look at which place leaks.

+ 1, there is a memory leak.

>
>
> regards
>
> --
> Tomas Vondra
>
>

So, I did some testing, and it seems that the tuple returned by
`gistgetadjusted` inside `gist_check_page` is not being freed.

Trivial fix attached.

-- 
Best regards,
Kirill Reshke

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. amcheck: Fix posting tree checks in gin_index_check()

  2. amcheck: Fix parent key check in gin_index_check()

  3. amcheck: Fix checks of entry order for GIN indexes

  4. amcheck: Remove unused GinScanItem->parentlsn field

  5. amcheck: Test gin_index_check on a multicolumn index

  6. Remove incidental md5() function use from test

  7. amcheck: Add a GIN index to the CREATE INDEX CONCURRENTLY tests

  8. amcheck: Add a test with GIN index on JSONB data

  9. amcheck: Fix indentation in verify_gin.c

  10. amcheck: Add gin_index_check() to verify GIN index

  11. amcheck: Move common routines into a separate module

  12. Fix grammar in GIN README

  13. Avoid amcheck inline compression false positives.