Horton

Re: JIT compiling with LLVM v11

Catalin Iacob <iacobcatalin@gmail.com>

View thread View on postgr.es
From: Catalin Iacob <iacobcatalin@gmail.com>
To: Andres Freund <andres@anarazel.de>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Thomas Munro <thomas.munro@enterprisedb.com>, Pg Hackers <pgsql-hackers@postgresql.org>, Pierre Ducroquet <p.psql@pinaraf.info>
Date: 2018-03-15T18:14:09Z
Lists: pgsql-hackers 
On Thu, Mar 15, 2018 at 6:19 PM, Andres Freund <andres@anarazel.de> wrote:
> What we were talking about in this subthread was about a depency on
> clang, not LLVM. And that's just needed at buildtime, to generate the
> bitcode files (including synchronizing types / function signatures).

I was actually thinking of both the buildtime and runtime dependency
because I did not realize the PGDG packages already depend on EPEL.

> For the yum.pg.o, which already depends on EPEL, there's a new enough
> LLVM version.  There's a new enough version in RHEL proper, but it
> appears to only be there for mesa (llvm-private).

Indeed RHEL 7 comes with llvm-private for mesa but that doesn't seem
kosher to use for other things.

When I said packagers I was only thinking of PGDG. I was thinking the
software collections would be the likely solution for the PGDG
packages for both buildtime and runtime. But it seems using clang from
software collections and LLVM from EPEL is also a possibility,
assuming that the newer clang generates IR that the older libraries
are guaranteed to be able to load.

For RHEL proper, I would guess that PG11 is too late for RHEL8 which,
according to history, should be coming soon.

For RHEL9 I would really expect RedHat to add llvm and clang to proper
RHEL and build/run against those, even if they add it only for
Postgres (like they did for mesa). I really don't see them shipping
without a major speedup for a major DB, also because in the meantime
the JIT in PG will have matured. That's also why I find it important
to support gcc and not restrict JIT to clang builds as I expect that
RedHat and all other Linux distros want to build everything with gcc
and asking them to switch to clang or give up JIT will put them in a
hard spot. As far as I know clang does promise gcc compatibility in
the sense that one can link together .o files compiled with both so I
expect the combination not to cause issues (assuming the other
compiler flags affecting binary compatibility are aligned).

Commits

  1. Improve JIT docs.

  2. Add documentation for the JIT feature.

  3. Add EXPLAIN support for JIT.

  4. Add inlining support to LLVM JIT provider.

  5. JIT tuple deforming in LLVM JIT provider.

  6. Add FIELDNO_* macro designating offset into structs required for JIT.

  7. Add expression compilation support to LLVM JIT provider.

  8. Expand list of synchronized types and functions in LLVM JIT provider.

  9. Add helpers for emitting LLVM IR.

  10. Basic planner and executor integration for JIT.

  11. Debugging and profiling support for LLVM JIT provider.

  12. Support for optimizing and emitting code in LLVM JIT provider.

  13. Add file containing extensions of the LLVM C API.

  14. Basic JIT provider and error handling infrastructure.

  15. Add configure infrastructure (--with-llvm) to enable LLVM support.

  16. Add C++ support to configure.

  17. Add PGAC_PROG_VARCC_VARFLAGS_OPT autoconf macro.

  18. Fix VM buffer pin management in heap_lock_updated_tuple_rec().

  19. Allow tupleslots to have a fixed tupledesc, use in executor nodes.

  20. Expression evaluation based aggregate transition invocation.

  21. Perform slot validity checks in a separate pass over expression.

  22. Rely on executor utils to build targetlist for DML RETURNING.

  23. Refer to OS X as "macOS", except for the port name which is still "darwin".