Thread
Commits
-
Blindly attempt to fix sepgsql tests.
- 83bbcb04ab3b 10.0 landed
-
Faster expression evaluation and targetlist projection.
- b8d7f053c5c2 10.0 cited
-
pgsql: Faster expression evaluation and targetlist projection.
Andres Freund <andres@anarazel.de> — 2017-03-25T22:11:37Z
Faster expression evaluation and targetlist projection. This replaces the old, recursive tree-walk based evaluation, with non-recursive, opcode dispatch based, expression evaluation. Projection is now implemented as part of expression evaluation. This both leads to significant performance improvements, and makes future just-in-time compilation of expressions easier. The speed gains primarily come from: - non-recursive implementation reduces stack usage / overhead - simple sub-expressions are implemented with a single jump, without function calls - sharing some state between different sub-expressions - reduced amount of indirect/hard to predict memory accesses by laying out operation metadata sequentially; including the avoidance of nearly all of the previously used linked lists - more code has been moved to expression initialization, avoiding constant re-checks at evaluation time Future just-in-time compilation (JIT) has become easier, as demonstrated by released patches intended to be merged in a later release, for primarily two reasons: Firstly, due to a stricter split between expression initialization and evaluation, less code has to be handled by the JIT. Secondly, due to the non-recursive nature of the generated "instructions", less performance-critical code-paths can easily be shared between interpreted and compiled evaluation. The new framework allows for significant future optimizations. E.g.: - basic infrastructure for to later reduce the per executor-startup overhead of expression evaluation, by caching state in prepared statements. That'd be helpful in OLTPish scenarios where initialization overhead is measurable. - optimizing the generated "code". A number of proposals for potential work has already been made. - optimizing the interpreter. Similarly a number of proposals have been made here too. The move of logic into the expression initialization step leads to some backward-incompatible changes: - Function permission checks are now done during expression initialization, whereas previously they were done during execution. In edge cases this can lead to errors being raised that previously wouldn't have been, e.g. a NULL array being coerced to a different array type previously didn't perform checks. - The set of domain constraints to be checked, is now evaluated once during expression initialization, previously it was re-built every time a domain check was evaluated. For normal queries this doesn't change much, but e.g. for plpgsql functions, which caches ExprStates, the old set could stick around longer. The behavior around might still change. Author: Andres Freund, with significant changes by Tom Lane, changes by Heikki Linnakangas Reviewed-By: Tom Lane, Heikki Linnakangas Discussion: https://postgr.es/m/20161206034955.bh33paeralxbtluv@alap3.anarazel.de Branch ------ master Details ------- http://git.postgresql.org/pg/commitdiff/b8d7f053c5c2bf2a7e8734fe3327f6a8bc711755 Modified Files -------------- contrib/postgres_fdw/postgres_fdw.c | 2 +- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/index.c | 43 +- src/backend/catalog/partition.c | 3 +- src/backend/catalog/toasting.c | 2 +- src/backend/commands/analyze.c | 10 +- src/backend/commands/explain.c | 2 +- src/backend/commands/indexcmds.c | 4 +- src/backend/commands/prepare.c | 4 +- src/backend/commands/tablecmds.c | 25 +- src/backend/commands/trigger.c | 10 +- src/backend/executor/Makefile | 7 +- src/backend/executor/README | 176 +- src/backend/executor/execExpr.c | 2665 +++++++++++++++ src/backend/executor/execExprInterp.c | 3525 +++++++++++++++++++ src/backend/executor/execIndexing.c | 20 +- src/backend/executor/execMain.c | 39 +- src/backend/executor/execQual.c | 5313 ----------------------------- src/backend/executor/execSRF.c | 924 +++++ src/backend/executor/execScan.c | 10 +- src/backend/executor/execUtils.c | 338 +- src/backend/executor/functions.c | 2 +- src/backend/executor/nodeAgg.c | 34 +- src/backend/executor/nodeBitmapHeapscan.c | 17 +- src/backend/executor/nodeCtescan.c | 8 +- src/backend/executor/nodeCustom.c | 8 +- src/backend/executor/nodeForeignscan.c | 15 +- src/backend/executor/nodeFunctionscan.c | 19 +- src/backend/executor/nodeGather.c | 8 +- src/backend/executor/nodeGatherMerge.c | 8 +- src/backend/executor/nodeGroup.c | 12 +- src/backend/executor/nodeHash.c | 12 +- src/backend/executor/nodeHashjoin.c | 44 +- src/backend/executor/nodeIndexonlyscan.c | 15 +- src/backend/executor/nodeIndexscan.c | 24 +- src/backend/executor/nodeMergejoin.c | 33 +- src/backend/executor/nodeModifyTable.c | 38 +- src/backend/executor/nodeNestloop.c | 23 +- src/backend/executor/nodeProjectSet.c | 61 +- src/backend/executor/nodeResult.c | 16 +- src/backend/executor/nodeSamplescan.c | 17 +- src/backend/executor/nodeSeqscan.c | 8 +- src/backend/executor/nodeSubplan.c | 96 +- src/backend/executor/nodeSubqueryscan.c | 8 +- src/backend/executor/nodeTableFuncscan.c | 20 +- src/backend/executor/nodeTidscan.c | 139 +- src/backend/executor/nodeValuesscan.c | 10 +- src/backend/executor/nodeWindowAgg.c | 8 +- src/backend/executor/nodeWorktablescan.c | 8 +- src/backend/optimizer/path/costsize.c | 2 +- src/backend/optimizer/plan/planner.c | 2 +- src/backend/optimizer/util/clauses.c | 12 +- src/backend/utils/adt/domains.c | 29 +- src/backend/utils/adt/ruleutils.c | 4 +- src/backend/utils/adt/xml.c | 45 +- src/backend/utils/cache/typcache.c | 40 +- src/include/executor/execExpr.h | 642 ++++ src/include/executor/execdebug.h | 23 +- src/include/executor/executor.h | 178 +- src/include/executor/nodeSubplan.h | 4 + src/include/fmgr.h | 8 + src/include/nodes/execnodes.h | 504 +-- src/include/nodes/nodes.h | 30 +- src/include/utils/typcache.h | 3 +- src/include/utils/xml.h | 4 +- src/pl/plpgsql/src/pl_exec.c | 5 +- src/test/regress/expected/case.out | 2 +- src/test/regress/expected/privileges.out | 20 +- src/test/regress/sql/case.sql | 2 +- src/test/regress/sql/privileges.sql | 5 +- src/tools/pgindent/typedefs.list | 6 +- 71 files changed, 8871 insertions(+), 6534 deletions(-)
-
Re: pgsql: Faster expression evaluation and targetlist projection.
Thomas Munro <thomas.munro@enterprisedb.com> — 2017-03-25T22:55:13Z
On Sun, Mar 26, 2017 at 11:11 AM, Andres Freund <andres@anarazel.de> wrote: > Faster expression evaluation and targetlist projection. > > This replaces the old, recursive tree-walk based evaluation, with > non-recursive, opcode dispatch based, expression evaluation. > Projection is now implemented as part of expression evaluation. > > This both leads to significant performance improvements, and makes > future just-in-time compilation of expressions easier. This is a huge achievement. Congratulations! -- Thomas Munro http://www.enterprisedb.com
-
Re: pgsql: Faster expression evaluation and targetlist projection.
Peter Geoghegan <pg@bowt.ie> — 2017-03-25T22:56:58Z
On Sat, Mar 25, 2017 at 3:55 PM, Thomas Munro <thomas.munro@enterprisedb.com> wrote: > This is a huge achievement. Congratulations! I also think it's excellent. Well done to all involved. -- Peter Geoghegan
-
Re: pgsql: Faster expression evaluation and targetlist projection.
David Rowley <david.rowley@2ndquadrant.com> — 2017-03-25T22:58:55Z
On 26 March 2017 at 09:55, Thomas Munro <thomas.munro@enterprisedb.com> wrote: > On Sun, Mar 26, 2017 at 11:11 AM, Andres Freund <andres@anarazel.de> wrote: >> Faster expression evaluation and targetlist projection. >> >> This replaces the old, recursive tree-walk based evaluation, with >> non-recursive, opcode dispatch based, expression evaluation. >> Projection is now implemented as part of expression evaluation. >> >> This both leads to significant performance improvements, and makes >> future just-in-time compilation of expressions easier. > > This is a huge achievement. Congratulations! Agreed. Congratulations! -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
-
Re: [COMMITTERS] pgsql: Faster expression evaluation and targetlist projection.
Andres Freund <andres@anarazel.de> — 2017-03-25T23:03:00Z
Hi Joe, On 2017-03-25 22:11:37 +0000, Andres Freund wrote: > Faster expression evaluation and targetlist projection. Apparently this broke the selinux integration somehow: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=rhinoceros&dt=2017-03-25%2022%3A45%3A01 Unfortunately the buildfarm appears to not display regression.diffs - making this a bit hard to debug remotely. Any chance you could help me out with that? I'd rather not setup selinux here... Regards, Andres
-
Re: [COMMITTERS] pgsql: Faster expression evaluation and targetlist projection.
Joe Conway <mail@joeconway.com> — 2017-03-25T23:36:03Z
On 03/25/2017 04:03 PM, Andres Freund wrote: > Hi Joe, > > > On 2017-03-25 22:11:37 +0000, Andres Freund wrote: >> Faster expression evaluation and targetlist projection. > > Apparently this broke the selinux integration somehow: > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=rhinoceros&dt=2017-03-25%2022%3A45%3A01 > > Unfortunately the buildfarm appears to not display regression.diffs - > making this a bit hard to debug remotely. Any chance you could help me > out with that? I'd rather not setup selinux here... Sure. What exactly do you want? Should we jump on a call or hangout or something? -- Crunchy Data - http://crunchydata.com PostgreSQL Support for Secure Enterprises Consulting, Training, & Open Source Development
-
Re: [COMMITTERS] pgsql: Faster expression evaluation and targetlist projection.
Andres Freund <andres@anarazel.de> — 2017-03-25T23:45:33Z
On 2017-03-25 16:36:03 -0700, Joe Conway wrote: > On 03/25/2017 04:03 PM, Andres Freund wrote: > > Hi Joe, > > > > > > On 2017-03-25 22:11:37 +0000, Andres Freund wrote: > >> Faster expression evaluation and targetlist projection. > > > > Apparently this broke the selinux integration somehow: > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=rhinoceros&dt=2017-03-25%2022%3A45%3A01 > > > > Unfortunately the buildfarm appears to not display regression.diffs - > > making this a bit hard to debug remotely. Any chance you could help me > > out with that? I'd rather not setup selinux here... > > > Sure. What exactly do you want? I think, for starters, seeing regression.diffs from sepgsql would be useful. Might already clear up what's the issue. > Should we jump on a call or hangout or something? I think for now just seeing regression.diffs would be enough - depending on what the differences are. Could very well be that we're just seing a difference due to function permission checks happening a bit earlier now. Greetings, Andres Freund
-
Re: pgsql: Faster expression evaluation and targetlist projection.
Michael Paquier <michael.paquier@gmail.com> — 2017-03-25T23:53:50Z
On Sun, Mar 26, 2017 at 7:58 AM, David Rowley <david.rowley@2ndquadrant.com> wrote: > On 26 March 2017 at 09:55, Thomas Munro <thomas.munro@enterprisedb.com> wrote: >> On Sun, Mar 26, 2017 at 11:11 AM, Andres Freund <andres@anarazel.de> wrote: >>> Faster expression evaluation and targetlist projection. >>> >>> This replaces the old, recursive tree-walk based evaluation, with >>> non-recursive, opcode dispatch based, expression evaluation. >>> Projection is now implemented as part of expression evaluation. >>> >>> This both leads to significant performance improvements, and makes >>> future just-in-time compilation of expressions easier. >> >> This is a huge achievement. Congratulations! > > Agreed. Congratulations! Hurray. Congratulations. -- Michael
-
Re: [COMMITTERS] pgsql: Faster expression evaluation and targetlist projection.
Joe Conway <mail@joeconway.com> — 2017-03-25T23:54:08Z
On 03/25/2017 04:45 PM, Andres Freund wrote: > On 2017-03-25 16:36:03 -0700, Joe Conway wrote: >> On 03/25/2017 04:03 PM, Andres Freund wrote: >> > Hi Joe, >> > >> > >> > On 2017-03-25 22:11:37 +0000, Andres Freund wrote: >> >> Faster expression evaluation and targetlist projection. >> > >> > Apparently this broke the selinux integration somehow: >> > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=rhinoceros&dt=2017-03-25%2022%3A45%3A01 >> > >> > Unfortunately the buildfarm appears to not display regression.diffs - >> > making this a bit hard to debug remotely. Any chance you could help me >> > out with that? I'd rather not setup selinux here... >> >> >> Sure. What exactly do you want? > > I think, for starters, seeing regression.diffs from sepgsql would be > useful. Might already clear up what's the issue. I went looking, and even after a forced run the diff file is gone -- does the buildfarm auto-cleanup or something? Joe -- Crunchy Data - http://crunchydata.com PostgreSQL Support for Secure Enterprises Consulting, Training, & Open Source Development
-
Re: Re: [COMMITTERS] pgsql: Faster expression evaluation and targetlist projection.
Andres Freund <andres@anarazel.de> — 2017-03-26T00:21:18Z
On March 25, 2017 4:54:08 PM PDT, Joe Conway <mail@joeconway.com> wrote: >On 03/25/2017 04:45 PM, Andres Freund wrote: >> On 2017-03-25 16:36:03 -0700, Joe Conway wrote: >>> On 03/25/2017 04:03 PM, Andres Freund wrote: >>> > Hi Joe, >>> > >>> > >>> > On 2017-03-25 22:11:37 +0000, Andres Freund wrote: >>> >> Faster expression evaluation and targetlist projection. >>> > >>> > Apparently this broke the selinux integration somehow: >>> > >https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=rhinoceros&dt=2017-03-25%2022%3A45%3A01 >>> > >>> > Unfortunately the buildfarm appears to not display >regression.diffs - >>> > making this a bit hard to debug remotely. Any chance you could >help me >>> > out with that? I'd rather not setup selinux here... >>> >>> >>> Sure. What exactly do you want? >> >> I think, for starters, seeing regression.diffs from sepgsql would be >> useful. Might already clear up what's the issue. > >I went looking, and even after a forced run the diff file is gone -- >does the buildfarm auto-cleanup or something? Yes, it does. You'd probably have to run the tests manually. Do you have selinux setup? I assumed you would, given I seen to recall a talk of yours with references to it ;) -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
-
Re: Re: [COMMITTERS] pgsql: Faster expression evaluation and targetlist projection.
Joe Conway <mail@joeconway.com> — 2017-03-26T00:33:33Z
On 03/25/2017 05:21 PM, Andres Freund wrote: > On March 25, 2017 4:54:08 PM PDT, Joe Conway <mail@joeconway.com> wrote: >>On 03/25/2017 04:45 PM, Andres Freund wrote: >>> I think, for starters, seeing regression.diffs from sepgsql would be >>> useful. Might already clear up what's the issue. >> >>I went looking, and even after a forced run the diff file is gone -- >>does the buildfarm auto-cleanup or something? > > Yes, it does. You'd probably have to run the tests manually. Do you > have selinux setup? I assumed you would, given I seen to recall a > talk of yours with references to it ;) Yeah, but those machines are MLS fully constrained, and the sepgsql regression test specifically needs "targeted" and some other particular setup. So the easiest box to run this on is the buildfarm animal, but I also want to do it in a way that doesn't mess up that environment. I found "keep_error_builds" in build-farm.conf and tried setting to 1 and rerunning in force -- that seems to have worked, so diffs attached. Joe -- Crunchy Data - http://crunchydata.com PostgreSQL Support for Secure Enterprises Consulting, Training, & Open Source Development
-
Re: Re: [COMMITTERS] pgsql: Faster expression evaluation and targetlist projection.
Andres Freund <andres@anarazel.de> — 2017-03-26T03:40:23Z
Hi, On 2017-03-25 17:33:33 -0700, Joe Conway wrote: > On 03/25/2017 05:21 PM, Andres Freund wrote: > > On March 25, 2017 4:54:08 PM PDT, Joe Conway <mail@joeconway.com> wrote: > >>On 03/25/2017 04:45 PM, Andres Freund wrote: > >>> I think, for starters, seeing regression.diffs from sepgsql would be > >>> useful. Might already clear up what's the issue. > >> > >>I went looking, and even after a forced run the diff file is gone -- > >>does the buildfarm auto-cleanup or something? > > > > Yes, it does. You'd probably have to run the tests manually. Do you > > have selinux setup? I assumed you would, given I seen to recall a > > talk of yours with references to it ;) > Yeah, but those machines are MLS fully constrained, and the sepgsql > regression test specifically needs "targeted" and some other particular > setup. So the easiest box to run this on is the buildfarm animal, but I > also want to do it in a way that doesn't mess up that environment. Ah. > I found "keep_error_builds" in build-farm.conf and tried setting to 1 > and rerunning in force -- that seems to have worked, so diffs attached. Thanks. Those all look like "valid" differences, i.e. checks that previously hadn't been executed because the function is never invoked, as there's no data in those tables. I blindly tried to fix these, let's hope that works. - Andres
-
Re: Re: [COMMITTERS] pgsql: Faster expression evaluation and targetlist projection.
Andres Freund <andres@anarazel.de> — 2017-03-26T04:52:39Z
On 2017-03-25 20:40:23 -0700, Andres Freund wrote: > I blindly tried to fix these, let's hope that works. In a second attempt (yes, reading diffs correctly is helpful), this resolved the selinux issue. Yeha. - Andres
-
Re: Re: [COMMITTERS] pgsql: Faster expression evaluation and targetlist projection.
Joe Conway <mail@joeconway.com> — 2017-03-26T15:30:47Z
On 03/25/2017 09:52 PM, Andres Freund wrote: > On 2017-03-25 20:40:23 -0700, Andres Freund wrote: >> I blindly tried to fix these, let's hope that works. > > In a second attempt (yes, reading diffs correctly is helpful), this > resolved the selinux issue. Yeha. +1! Joe -- Crunchy Data - http://crunchydata.com PostgreSQL Support for Secure Enterprises Consulting, Training, & Open Source Development