Horton

JIT compiling with LLVM v11

Andres Freund <andres@anarazel.de>

View thread View on postgr.es
From: Andres Freund <andres@anarazel.de>
To: pgsql-hackers@postgresql.org
Cc: Pierre Ducroquet <p.psql@pinaraf.info>
Date: 2018-03-01T08:02:42Z
Lists: pgsql-hackers 
Hi,

I've pushed a revised version of my JIT patchset.
The git tree is at
  https://git.postgresql.org/git/users/andresfreund/postgres.git
in the jit branch
  https://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git;a=shortlog;h=refs/heads/jit

Biggest changes:
- LLVM 3.9 - master are now supported. This includes a good chunk of
  work by Pierre Ducroquet.

  Doing so I found that the patches Pierre provided didn't work when a
  query was expensive enough to warrant inlining. Turns out LLVM < 5
  can't combine the summaries of multiple thin module summaries.

  But that actually turned out to be a good thing, because it made me
  think about symbol resolution preferences. Previously it was basically
  arbitrary whether a function with conflicting names would be choosen
  from core postgres or one of the extension libs providing it.

  This is now rewritten so we don't build a combined module summary for
  core postgres and extensions at backend start. Instead summaries for
  core pg and extensions are loaded separately, and the correct one for
  a symbol is used.

- Functions in extension libraries are now not referred to with their C
  symbol in LLVM IR, instead we generate a fictious symbol that includes
  the library path. E.g. hstore's hstore_from_record is now referenced as
@"pgextern.$libdir/hstore.hstore_from_record".

  Both symbol resolution and inlining knows how to properly resolve
  those.

- As hinted to above, the inlining support has evolved
  considerably. Instead of a combined index built at backend start we
  now have individual indexes for each extension / shlib.  Symbols are
  searched with a search path (internal functions just in the 'postgres'
  index, for extensions it's main 'postgres', 'extension'), and symbols
  that explicitly reference a function are just looked up within that
  extension.

  This has the nice advantage that we don't have to process indexes for
  extensions that aren't used, which in turn means that extensions can
  be installed on the system level while a backend is running, and
  JITing will work even for old backends once the extension is created
  (or rather functions in it).

  Additionally the inline costing logic has improved, the super verbose
  logging is #ifdef'ed out ('ilog' wrapper that's just (void) 0).

- The installation of bitcode is now a nice separate make function. pgxs
  (including contrib's kinda use of pgxs) now automatically generate &
  install bitcode when the server was compiled --with-llvm.

  I learned some about make I didn't know ;).

- bunch of compilation issues (older clang, -D_NDEBUG from llvm-config
  being used for all of postgres, ...) have been fixed.

- Two bigger prerequisite patches have been merged.

- lotsa smaller stuff.


Regards,

Andres

Commits

  1. Improve JIT docs.

  2. Add documentation for the JIT feature.

  3. Add EXPLAIN support for JIT.

  4. Add inlining support to LLVM JIT provider.

  5. JIT tuple deforming in LLVM JIT provider.

  6. Add FIELDNO_* macro designating offset into structs required for JIT.

  7. Add expression compilation support to LLVM JIT provider.

  8. Expand list of synchronized types and functions in LLVM JIT provider.

  9. Add helpers for emitting LLVM IR.

  10. Basic planner and executor integration for JIT.

  11. Debugging and profiling support for LLVM JIT provider.

  12. Support for optimizing and emitting code in LLVM JIT provider.

  13. Add file containing extensions of the LLVM C API.

  14. Basic JIT provider and error handling infrastructure.

  15. Add configure infrastructure (--with-llvm) to enable LLVM support.

  16. Add C++ support to configure.

  17. Add PGAC_PROG_VARCC_VARFLAGS_OPT autoconf macro.

  18. Fix VM buffer pin management in heap_lock_updated_tuple_rec().

  19. Allow tupleslots to have a fixed tupledesc, use in executor nodes.

  20. Expression evaluation based aggregate transition invocation.

  21. Perform slot validity checks in a separate pass over expression.

  22. Rely on executor utils to build targetlist for DML RETURNING.

  23. Refer to OS X as "macOS", except for the port name which is still "darwin".