Horton
Menu

Thread

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Doc: further tweaking of v14 release notes.

  1. Query planning on partitioned table causes postgres 13.4 to consume all memory

    Duncan Sands <duncan.sands@deepbluecap.com> — 2021-09-17T09:49:06Z

    Postgresql version: 13.4
O/S version: Ubuntu 21.04

How to reproduce:
   (1) Create "data" schema and its tables by executing the commands from the 
attached file d.sql.  Each of the two created tables has less than 20 rows.
   (2) Execute or EXPLAIN this query

DELETE FROM data.files

   WHERE path=ANY(

    SELECT path

      FROM (

             SELECT meta, path,

                first_value(path)

                   OVER (PARTITION BY meta ORDER BY priority) AS first

             FROM data.files

          ) f,

          data.metadata m

      WHERE f.path <> f.first

       AND f.meta = m.id

       AND m.syd = 667

    );


   (3) Observe that the EXPLAIN/query never completes, and memory usage goes up 
and up.


For example:

duncan=> \i /tmp/d.sql

...
duncan=> DELETE FROM data.files

   WHERE path=ANY(

    SELECT path

      FROM (

             SELECT meta, path,

                first_value(path)

                   OVER (PARTITION BY meta ORDER BY priority) AS first

             FROM data.files

          ) f,

          data.metadata m

      WHERE f.path <> f.first

       AND f.meta = m.id

       AND m.syd = 667

    );

... wailing and gnashing of teeth ...

  2. Re: Query planning on partitioned table causes postgres 13.4 to consume all memory

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-09-19T16:03:04Z

    Duncan Sands <duncan.sands@deepbluecap.com> writes:
> [ planning DELETE on a thousand-partition table takes forever ]

FWIW, this situation has been very much improved for v14 [1].
In older branches, the best advice I can give you is "don't use
so many partitions".  Especially not with hash partitioning,
where the query WHERE clause generally won't translate to any
useful pruning of the partitions.

(Personally, I think that hash partitioning is an evil that
we shouldn't have implemented at all.  Or at least there
should be stronger warnings about it in the manual than there
are now.)

			regards, tom lane

[1] https://git.postgresql.org/gitweb/?p=postgresql.git&a=commitdiff&h=86dc90056

  3. Re: Query planning on partitioned table causes postgres 13.4 to consume all memory

    Duncan Sands <duncan.sands@deepbluecap.com> — 2021-09-20T12:10:17Z

    Hi Tom,

On 19/09/2021 18:03, Tom Lane wrote:
> Duncan Sands <duncan.sands@deepbluecap.com> writes:
>> [ planning DELETE on a thousand-partition table takes forever ]
> 
> FWIW, this situation has been very much improved for v14 [1].

thanks, part (2) of that commit indeed looks like it should solve it.

Best wishes, Duncan.

> In older branches, the best advice I can give you is "don't use
> so many partitions".  Especially not with hash partitioning,
> where the query WHERE clause generally won't translate to any
> useful pruning of the partitions.
> 
> (Personally, I think that hash partitioning is an evil that
> we shouldn't have implemented at all.  Or at least there
> should be stronger warnings about it in the manual than there
> are now.)
> 
> 			regards, tom lane
> 
> [1] https://git.postgresql.org/gitweb/?p=postgresql.git&a=commitdiff&h=86dc90056
>