Horton

Re: pg_stat_statements and "IN" conditions

Sami Imseih <samimseih@gmail.com>

View thread
From: Sami Imseih <samimseih@gmail.com>
To: Dmitry Dolgov <9erthalion6@gmail.com>
Cc: Julien Rouhaud <rjuju123@gmail.com>, Álvaro Herrera <alvherre@alvh.no-ip.org>, Kirill Reshke <reshkekirill@gmail.com>, Sergei Kornilov <sk@zsrv.org>, yasuo.honda@gmail.com, tgl@sss.pgh.pa.us, smithpb2250@gmail.com, vignesh21@gmail.com, michael@paquier.xyz, nathandbossart@gmail.com, stark.cfm@gmail.com, geidav.pg@gmail.com, marcos@f10.com.br, robertmhaas@gmail.com, david@pgmasters.net, pgsql-hackers@postgresql.org, pavel.trukhanov@gmail.com, Sutou Kouhei <kou@clear-code.com>
Date: 2025-02-14T16:02:39Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Introduce squashing of constant lists in query jumbling

  2. Make documentation builds reproducible

  3. Include values of A_Const nodes in query jumbling

  4. Teach planner about more monotonic window functions

  5. Split up guc.c for better build speed and ease of maintenance.

> > I perhap meant "missing chunk" instead of "trimming". To me it just
> > looked like a trimmed text, which was wrong. Looks like v25
> > deals with that better at least. I am just not sure about all that we are doing
> > here as I believe it may open up big changes for bugs generating the normalized
> > query texts. I'm a bit worried about that. IMO, we are better off just
> > adding a comment
> > at the start of a query that this query text such as "/*
> > query_id_squash_values */"
> > and keeping all the parameter symbols in-place.
>
> I see what you mean, but keeping everything in place is partially
> defeating purpose of the patch. The idea is not only to make those
> queries to have the same query_id, but also to reduce the size of
> queries themselves. E.g. the use case scenario that has triggered the
> patch was about queries having dozens of thousands of such constants,
> so that the size of them was a burden on its own.

My experience with this issue is not so much the size of the query text,
but it's the fact that similar queries ( with varying length IN-lists ) being
tracked in different entries, causing high deallocation and heavy
garbage collection. This is besides the overall loss of quality of
the data from pg_stat_statements if there is constant deallocation.

But, with what you are doing with this patch, we will now have
a single tracking entry for similar queries with varying IN-lists and
even if the query text is *large*, it's only a single entry tracking
and we are no longer continuously deallocating and garbage
collecting as frequently.

-- 
Sami