Horton

Thread

Commits

  1. Reword doc blurb for vacuumdb --analyze-in-stages

  1. vacuumdb --analyze-in-stages

    nikolai.berkoff <nikolai.berkoff@pm.me> — 2021-10-08T09:33:51Z

    Running --analyze-in-stages on a database with statistics causes the statistics to become significantly worse before they are improved.  This could be made clearer in the documentation.

  2. Re: vacuumdb --analyze-in-stages

    Euler Taveira <euler@eulerto.com> — 2021-10-18T18:12:34Z

    On Fri, Oct 8, 2021, at 6:33 AM, nikolai.berkoff wrote:
> Running --analyze-in-stages on a database with statistics causes the statistics to become significantly worse before they are improved.  This could be made clearer in the documentation.
I think the paragraph you modified *already* stated that usefulness of
--analyze-in-stages (newly populated from a restored dump or by pg_upgrade). I
don't think your change is an improvement for --analyze-in-stages description.
If you read the previous paragraph, it explains _how_ (3 stages of ANALYZE with
different settings) and _why_ (produce usable statistics faster).


--
Euler Taveira
EDB   https://www.enterprisedb.com/

  3. Re: vacuumdb --analyze-in-stages

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-10-18T23:02:11Z

    On 2021-Oct-18, Euler Taveira wrote:

> On Fri, Oct 8, 2021, at 6:33 AM, nikolai.berkoff wrote:
> > Running --analyze-in-stages on a database with statistics causes the
> > statistics to become significantly worse before they are improved.
> > This could be made clearer in the documentation.
>
> I think the paragraph you modified *already* stated that usefulness of
> --analyze-in-stages (newly populated from a restored dump or by pg_upgrade). I
> don't think your change is an improvement for --analyze-in-stages description.
> If you read the previous paragraph, it explains _how_ (3 stages of ANALYZE with
> different settings) and _why_ (produce usable statistics faster).

Yeah, but it doesn't say that the initial stats target (1) is the worst
value you could possibly have.

I wonder if it isn't better to *remove* the second phrase from this
paragraph, and leave just this text:

        This option is useful to analyze a database that was newly populated
        from a restored dump or by <command>pg_upgrade</command>.

to avoid giving the impression that it'll improve things if you run it
in a database with existing statistics.  Or we could proactively warn
more explicitly about the problem:

        This option is useful to analyze a database that was newly populated
        from a restored dump or by <command>pg_upgrade</command>.
	Beware that running with this option in a database with existing
	statistics may cause query optimizer choices to become
	transiently worse, because of the very low statistics target
	that is used in the early stages.

Or maybe just

        This option is useful to analyze a database that was newly populated
        from a restored dump or by <command>pg_upgrade</command>.
	Running with this option in a database with existing statistics
	is not recommended.

Given that the first stage uses statistic target=1, running this option
in a database with any stats at all is probably a bad idea.

-- 
Álvaro Herrera           39°49'30"S 73°17'W  —  https://www.EnterpriseDB.com/

  4. Re: vacuumdb --analyze-in-stages

    David G. Johnston <david.g.johnston@gmail.com> — 2021-10-18T23:34:25Z

    On Mon, Oct 18, 2021 at 4:02 PM Alvaro Herrera <alvherre@alvh.no-ip.org>
wrote:

> On 2021-Oct-18, Euler Taveira wrote:
>
> > On Fri, Oct 8, 2021, at 6:33 AM, nikolai.berkoff wrote:
> > > Running --analyze-in-stages on a database with statistics causes the
> > > statistics to become significantly worse before they are improved.
> > > This could be made clearer in the documentation.
> >
> > I think the paragraph you modified *already* stated that usefulness of
> > --analyze-in-stages (newly populated from a restored dump or by
> pg_upgrade). I
> > don't think your change is an improvement for --analyze-in-stages
> description.
> > If you read the previous paragraph, it explains _how_ (3 stages of
> ANALYZE with
> > different settings) and _why_ (produce usable statistics faster).
>
> Yeah, but it doesn't say that the initial stats target (1) is the worst
> value you could possibly have.
> [...]
> Given that the first stage uses statistic target=1, running this option
> in a database with any stats at all is probably a bad idea.
>
>
Add the word "only"?

This option is only useful to analyze a database...

Beyond that maybe adjust the procedure description to include a comment
that we don't actually skip tables that already have a higher statistics
target than the current pass would apply. (can we do this?)

"Run several (currently three) stages of analyze with different
configuration settings, to produce usable statistics faster.  The first of
these stages will remove any existing statistics even if they use a larger
statistic target configuration."

David J.

  5. Re: vacuumdb --analyze-in-stages

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-10-18T23:53:52Z

    On 2021-Oct-18, David G. Johnston wrote:

> On Mon, Oct 18, 2021 at 4:02 PM Alvaro Herrera <alvherre@alvh.no-ip.org>
> wrote:

> > Given that the first stage uses statistic target=1, running this option
> > in a database with any stats at all is probably a bad idea.
>
> Add the word "only"?
> 
> This option is only useful to analyze a database...

Maybe this is sufficient, since it would drive people away from trying
to do anything else than help upgrades with it.

> Beyond that maybe adjust the procedure description to include a comment
> that we don't actually skip tables that already have a higher statistics
> target than the current pass would apply. (can we do this?)

Hmm, vacuumdb can certainly query the catalogs to see what we have and
skip tables for which we have more than that number, perhaps with a
query based on this number

select tablename,
    max(coalesce(cardinality(histogram_bounds), cardinality(most_common_freqs))) - 1
from pg_stats
group by tablename;

(and of course *don't* skip tables in the final stage, since the stored
stats could be obsolete.)

> "Run several (currently three) stages of analyze with different
> configuration settings, to produce usable statistics faster.  The first of
> these stages will remove any existing statistics even if they use a larger
> statistic target configuration."

.. yeah, this is another option.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/

  6. Re: vacuumdb --analyze-in-stages

    Euler Taveira <euler@eulerto.com> — 2021-10-19T00:02:08Z

    On Mon, Oct 18, 2021, at 8:53 PM, Alvaro Herrera wrote:
> On 2021-Oct-18, David G. Johnston wrote:
> 
> > On Mon, Oct 18, 2021 at 4:02 PM Alvaro Herrera <alvherre@alvh.no-ip.org>
> > wrote:
> 
> > > Given that the first stage uses statistic target=1, running this option
> > > in a database with any stats at all is probably a bad idea.
> >
> > Add the word "only"?
> > 
> > This option is only useful to analyze a database...
> 
> Maybe this is sufficient, since it would drive people away from trying
> to do anything else than help upgrades with it.
+1. I like your 2nd suggestion.

"This option is only useful to analyze a database that was newly populated from
a restored dump or by <command>pg_upgrade</command>.  Beware that running with
this option in a database with existing statistics may cause query optimizer
choices to become transiently worse, because of the very low statistics target
that is used in the early stages."

> 
> > "Run several (currently three) stages of analyze with different
> > configuration settings, to produce usable statistics faster.  The first of
> > these stages will remove any existing statistics even if they use a larger
> > statistic target configuration."
> 
> .. yeah, this is another option.
We might include it too but I would suggest "replace" instead of "remove"
because it seems there won't be statistics after the first stage.


--
Euler Taveira
EDB   https://www.enterprisedb.com/

  7. Re: vacuumdb --analyze-in-stages

    nikolai.berkoff <nikolai.berkoff@pm.me> — 2021-10-26T14:18:51Z

    ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Tuesday, October 19th, 2021 at 01:02, Euler Taveira <euler@eulerto.com> wrote:

> On Mon, Oct 18, 2021, at 8:53 PM, Alvaro Herrera wrote:
> 

> > On 2021-Oct-18, David G. Johnston wrote:
> > 

> > > On Mon, Oct 18, 2021 at 4:02 PM Alvaro Herrera <alvherre@alvh.no-ip.org>
> > > wrote:
> > 

> > > > Given that the first stage uses statistic target=1, running this option
> > > > in a database with any stats at all is probably a bad idea.
> > >
> > > Add the word "only"?
> > > 
> > > This option is only useful to analyze a database...
> > 

> > Maybe this is sufficient, since it would drive people away from trying
> > to do anything else than help upgrades with it.
> 

> +1. I like your 2nd suggestion.
> 

> "This option is only useful to analyze a database that was newly populated from
> a restored dump or by <command>pg_upgrade</command>.  Beware that running with
> this option in a database with existing statistics may cause query optimizer
> choices to become transiently worse, because of the very low statistics target
> that is used in the early stages."
> 

> > > "Run several (currently three) stages of analyze with different
> > > configuration settings, to produce usable statistics faster.  The first of
> > > these stages will remove any existing statistics even if they use a larger
> > > statistic target configuration."
> > 

> > .. yeah, this is another option.
> 

> We might include it too but I would suggest "replace" instead of "remove"
> because it seems there won't be statistics after the first stage.

Given all the suggestions I've tried to combine them into one patch again.

Regards,

Nikolai

  8. Re: vacuumdb --analyze-in-stages

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-11-05T14:25:25Z

    On 2021-Oct-26, nikolai.berkoff wrote:

> Given all the suggestions I've tried to combine them into one patch again.

OK, I pushed something very similar but not exactly your patch ...
please have a look if you care to suggest improvements.

... oh, I just noticed I made a typo -- omitted "with" in "running this
option".  I'll wait before pushing a fix in case there are other
rewording suggestions.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/