Thread

llvm dependency and space concerns

Jeremy Schneider <schneider@ardentperf.com> — 2025-01-11T20:56:19Z

I'm running Postgres in containers, and recently did some analysis of
the total container sizes. I posted some analysis over on the debian
packaging mailing list [1] [2]. The TLDR is that LLVM alone makes up
33% of a postgres container's bytes (143MB / 434MB) [1].

Per the details in the referenced emails, dpkg Installed-Size:
libllvm16     120542 KB
libz3-4       22767 KB

(Note that libz3 is a dependency of libllvm.)

For plperl, plpython and pltcl we are able to split those into separate
debian packages because Postgres' extension framework enables us to
install the files separately without a recompile, and Postgres can be
compiled with these configure flags but still works fine if the files
aren't present.

I haven't yet looked closely, but my assumption is that the --with-llvm
flag may not work the same. I'm going to spend some time taking a look,
but if someone knows off the top of their head and can give me a head
start that would be appreciated!

Given the large number of bytes that LLVM pulls into a postgres build,
I think it would be a good idea to have the ability to split it into a
separate [recommended, but optional] package. There are use cases like
embedded and IoT - in addition to containers - where some postgres users
value this level of space savings over JIT.

Thanks
-Jeremy Schneider

Re: llvm dependency and space concerns

Jeremy Schneider <schneider@ardentperf.com> — 2025-01-11T21:01:08Z

On Sat, 11 Jan 2025 12:56:19 -0800
Jeremy Schneider <schneider@ardentperf.com> wrote:

> I'm running Postgres in containers, and recently did some analysis of
> the total container sizes. I posted some analysis over on the debian
> packaging mailing list [1] [2]. The TLDR is that LLVM alone makes up
> 33% of a postgres container's bytes (143MB / 434MB) [1].
> 
> Per the details in the referenced emails, dpkg Installed-Size:
> libllvm16     120542 KB
> libz3-4       22767 KB

forgot the links :)

1: https://postgr.es/m/20250111122847.486d05e2%40jeremy-ThinkPad-T430s
2: https://postgr.es/m/20250109005301.3b145092%40jeremy-ThinkPad-T430s

Re: llvm dependency and space concerns

Tom Lane <tgl@sss.pgh.pa.us> — 2025-01-11T21:14:19Z

Jeremy Schneider <schneider@ardentperf.com> writes:
> Given the large number of bytes that LLVM pulls into a postgres build,
> I think it would be a good idea to have the ability to split it into a
> separate [recommended, but optional] package.

Build without --with-llvm.  Alternatively, split lib/llvmjit.so and
lib/bitcode/ into a separate package.  These are matters for packagers
not the core project ...

			regards, tom lane

Re: llvm dependency and space concerns

Jeremy Schneider <schneider@ardentperf.com> — 2025-01-11T21:22:39Z

On Sat, 11 Jan 2025 16:14:19 -0500
Tom Lane <tgl@sss.pgh.pa.us> wrote:

> Jeremy Schneider <schneider@ardentperf.com> writes:
> > Given the large number of bytes that LLVM pulls into a postgres
> > build, I think it would be a good idea to have the ability to split
> > it into a separate [recommended, but optional] package.  
> 
> Build without --with-llvm.  Alternatively, split lib/llvmjit.so and
> lib/bitcode/ into a separate package.  These are matters for packagers
> not the core project ...

This was my initial idea too, I was thinking to have a "normal"
postgres package and an additional a "slim" or "nojit" postgres package
(this was my original topic on the packagers list).

We would have to update dependencies for the dozens of packages like
extensions, wal2json, pgtop, and all the rest to take either of these
base postgres packages, and we have to deal with the two base packages
that potentially conflict with each other.

It's a cleaner solution if JIT works more like an extension, and we can
run a single build and split JIT into a separate package.

-Jeremy

Re: llvm dependency and space concerns

Christoph Moench-Tegeder <cmt@burggraben.net> — 2025-01-11T23:33:35Z

## Jeremy Schneider (schneider@ardentperf.com):

> I'm running Postgres in containers, and recently did some analysis of
> the total container sizes. I posted some analysis over on the debian
> packaging mailing list [1] [2]. The TLDR is that LLVM alone makes up
> 33% of a postgres container's bytes (143MB / 434MB) [1].

Does that even matter compared with the size of the database itself?

> I haven't yet looked closely, but my assumption is that the --with-llvm
> flag may not work the same. I'm going to spend some time taking a look,
> but if someone knows off the top of their head and can give me a head
> start that would be appreciated!

Have a look at the RPM packages and how they split off postgresql-llvmjit
there:
https://git.postgresql.org/gitweb/?p=pgrpms.git;a=tree;f=rpm/redhat/main/non-common/postgresql-17/main;hb=HEAD

Regards,
Christoph

-- 
Spare Space

Re: llvm dependency and space concerns

Andres Freund <andres@anarazel.de> — 2025-01-12T00:03:43Z

Hi,

On 2025-01-11 13:22:39 -0800, Jeremy Schneider wrote:
> It's a cleaner solution if JIT works more like an extension, and we can
> run a single build and split JIT into a separate package.

It does work like that. Only llvmjit.so has the llvm dependency, the main
postgres binary doesn't link to llvm. If llvmjit.so isn't available, jit is
silently disabled.

Andres

Re: llvm dependency and space concerns

Jeremy Schneider <schneider@ardentperf.com> — 2025-01-12T02:37:09Z

On Sat, 11 Jan 2025 19:03:43 -0500
Andres Freund <andres@anarazel.de> wrote:

> If llvmjit.so isn't available, jit is silently disabled.

Thanks - this is exactly the bit I wasn't sure about. The reason I came
to the hackers list is because I thought this wasn't the case. Normally
I don't delete arbitrary files after building postgres, and expect
postgres to continue working :)  Very happy to learn it works this way!

Also thanks to Christoph Moench-Tegeder who pointed out in the other
email that the PGDG rpm packagers already split LLVM out to a separate
package, so it's already been done over there.

-Jeremy