Horton

Thread

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Workaround code generation bug in clang

  2. Don't reduce output request size on non-Unix-socket connections.

  1. Invalid control file checksum with AVX-512 during initdb on a clang19 -O0 build

    Soumyadeep Chakraborty <soumyadeep2007@gmail.com> — 2025-06-11T18:10:31Z

    Hello,

I noticed a strange issue that I can only reproduce with clang:
clang version 19.1.7 (RESF 19.1.7-2.module+el8.10.0+1965+112b558b) on
the devel branch (SHA: 137935bd1167a94b0bfea7239033f1ba1a1d95bb).

We are getting a control file checksum mismatch during initdb. I added
some prints in a small debug patch, and recorded the postgres process
using rr. I have uploaded the rr archive (made with rr pack, tar-ed up) [1].

$ initdb -n -d -D /usr/local/pgsql/data &> initdb.out

2025-06-11 16:19:54.343 UTC [3070] LOG:  WriteControlFile crc = 3457434907,
algo is avx = 1
...
2025-06-11 16:19:54.343 UTC [3070] LOG:  ReadControlFile crc = 3457434907,
ControlFile->crc = 3457434907, algo is avx = 1
...
2025-06-11 16:19:54.346 UTC [3070] LOG:  update_controlfile crc =
2065009488, algo is avx = 1
...
2025-06-11 16:20:13.914 UTC [3070] LOG:  update_controlfile crc =
3406554082, algo is avx = 1
...
2025-06-11 16:20:13.920 UTC [3070] LOG:  update_controlfile crc =
1234673735, algo is avx = 1
...
2025-06-11 16:20:13.923 UTC [3070] NOTICE:  database system is shut down
...
2025-06-11 16:20:13.923 UTC [3070] DEBUG:  proc_exit(-1): 0 callbacks to
make
ok
performing post-bootstrap initialization ... 2025-06-11 16:20:13.984 UTC
[3072] LOG:  ReadControlFile crc = 2925279607, ControlFile->crc =
1234673735, algo is avx = 1
2025-06-11 16:20:13.984 UTC [3072] FATAL:  incorrect checksum in control
file
child process exited with exit code 1
initdb: data directory "/usr/local/pgsql/data" not removed at user's request

Note that this only reproduces with clang-19 -O0 and NOT -O3. I haven't
tried with other versions of clang.

OTOH, gcc-14 is cool with both -O0 and -O3, with AVX-512 getting picked
for both cases, for CRC instructions.

Environment:

(1) Configure options:
./configure --prefix=/usr/local/pgsql --with-python --enable-depend
--without-icu --enable-debug CFLAGS='-O0 -fno-omit-frame-pointer' CC=clang

(2) Config log shows:
configure:18262: checking for vectorized CRC-32C
configure:18268: result: AVX-512 with runtime check

pgac_cv_avx512_pclmul_intrinsics=yes
pgac_cv_xsave_intrinsics=yes

(3) Confirmation that we have AVX-512 CRC getting selected at runtime:
(rr) f
#0  WriteControlFile () at xlog.c:4386
4386 ControlFile->pg_control_version = PG_CONTROL_VERSION;
(rr) p pg_comp_crc32c
$1 = (pg_crc32c (*)(pg_crc32c, const void *, size_t)) 0xc8b8b0
<pg_comp_crc32c_avx512>

(4) This is running in a VM with:
Rocky Linux release 8.10 (Green Obsidian)
16 vCPUs
Hypervisor: VMware ESXi, 8.0.3, 24022510
Model: PowerEdge R650
Processor Type: Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz
vCenter:
Version: 8.0.3
Build: 24322831

I attempted both vSAN and local storage. That didn't influence matters.
There is a known vSAN bug with invalid checksums + AVX-512, but that has
been fixed in an older version [2] (and besides the issue reproes with
local storage too).

Please let me know if there is any other info I can provide.

Regards,
Deep (VMware)

[1]
https://drive.google.com/file/d/15bGN_NlGsYx0lJCMGnCxDnV2avYkujgk/view?usp=sharing
[2]
https://knowledge.broadcom.com/external/article/367589/applications-using-avx512-instructions-i.html

  2. Re: Invalid control file checksum with AVX-512 during initdb on a clang19 -O0 build

    Nathan Bossart <nathandbossart@gmail.com> — 2025-06-11T21:42:19Z

    (I've CC'd the committer of the feature.)

On Wed, Jun 11, 2025 at 11:10:31AM -0700, Soumyadeep Chakraborty wrote:
> We are getting a control file checksum mismatch during initdb.
> 
> [...]
> 
> Note that this only reproduces with clang-19 -O0 and NOT -O3. I haven't
> tried with other versions of clang.
> 
> OTOH, gcc-14 is cool with both -O0 and -O3, with AVX-512 getting picked
> for both cases, for CRC instructions.

I'm wondering if there's anything else this could be besides a compiler
bug.  It's interesting that -O0 is the broken case.  Have you tried looking
for differences in the assembly code?

-- 
nathan

  3. Re: Invalid control file checksum with AVX-512 during initdb on a clang19 -O0 build

    John Naylor <johncnaylorls@gmail.com> — 2025-06-12T06:57:01Z

    On Thu, Jun 12, 2025 at 4:42 AM Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> (I've CC'd the committer of the feature.)
>
> On Wed, Jun 11, 2025 at 11:10:31AM -0700, Soumyadeep Chakraborty wrote:
> > Note that this only reproduces with clang-19 -O0 and NOT -O3. I haven't
> > tried with other versions of clang.
> >
> > OTOH, gcc-14 is cool with both -O0 and -O3, with AVX-512 getting picked
> > for both cases, for CRC instructions.
>
> I'm wondering if there's anything else this could be besides a compiler
> bug.  It's interesting that -O0 is the broken case.  Have you tried looking
> for differences in the assembly code?

Hmm, clang at -O0 is the same in versions 18, 19, and 20. The change
from v17 is easy enough to workaround as in the attached, which
produces identical output as master at -O2. Does this fix it for you?
If that's not it, it may be more difficult, as the change at -O0 from
v16 to v17 is large, so it's hard to see what's going on.

-- 
John Naylor
Amazon Web Services

  4. Re: Invalid control file checksum with AVX-512 during initdb on a clang19 -O0 build

    Soumyadeep Chakraborty <soumyadeep2007@gmail.com> — 2025-06-12T17:30:08Z

    On Wed, Jun 11, 2025 at 11:57PM John Naylor <johncnaylorls@gmail.com> wrote:
>
> On Thu, Jun 12, 2025 at 4:42AM Nathan Bossart <nathandbossart@gmail.com>
wrote:
> >
> > (I've CC'd the committer of the feature.)
> >
> > On Wed, Jun 11, 2025 at 11:10:31AM -0700, Soumyadeep Chakraborty wrote:
> > > Note that this only reproduces with clang-19 -O0 and NOT -O3. I
haven't
> > > tried with other versions of clang.
> > >
> > > OTOH, gcc-14 is cool with both -O0 and -O3, with AVX-512 getting
picked
> > > for both cases, for CRC instructions.
> >
> > I'm wondering if there's anything else this could be besides a compiler
> > bug.  It's interesting that -O0 is the broken case.  Have you tried
looking
> > for differences in the assembly code?

Unfortunately, I haven't had a chance to dig into the assembly yet.

>
> Hmm, clang at -O0 is the same in versions 18, 19, and 20. The change
> from v17 is easy enough to workaround as in the attached, which
> produces identical output as master at -O2. Does this fix it for you?
> If that's not it, it may be more difficult, as the change at -O0 from
> v16 to v17 is large, so it's hard to see what's going on.
>

It unfortunately doesn't. I still get the same checksum failure. I tried
-O0 clang-19, clang-18 and clang-17. I've uploaded the rr recordings
and initdb output with your patch applied and the debug logs I had.

Regards,
Deep

 initdb-clang17.crc.workaround.out
<https://drive.google.com/file/d/1OptFhlWsOqNx4s2fv1tikOhsqjh27u-E/view?usp=drive_web>
 initdb-clang18.crc.workaround.out
<https://drive.google.com/file/d/1yXYWmt7cHMXgONMkr9Yl29YU2HXpE8LT/view?usp=drive_web>
 initdb-clang19.crc.workaround.out
<https://drive.google.com/file/d/1JmLKQMXOCwx8YFlJ_xpKdniopjwGpcji/view?usp=drive_web>
 rr-clang17.crc.workaround.tar.gz
<https://drive.google.com/file/d/1Tnjva4l4w6S6iKn2hJBaTy1NRNHTTsAV/view?usp=drive_web>
 rr-clang18.crc.workaround.tar.gz
<https://drive.google.com/file/d/1m7iKJHcAPPcKXet9gKkGEzTiFbJ2C9Ul/view?usp=drive_web>
 rr-clang19.crc.workaround.tar.gz
<https://drive.google.com/file/d/1zXzUtvMJbsNMJNQezI5uC3Fb8Y3sUuMs/view?usp=drive_web>

  5. Re: Invalid control file checksum with AVX-512 during initdb on a clang19 -O0 build

    John Naylor <johncnaylorls@gmail.com> — 2025-06-13T14:03:16Z

    On Fri, Jun 13, 2025 at 12:30 AM Soumyadeep Chakraborty <
soumyadeep2007@gmail.com> wrote:
>
> On Wed, Jun 11, 2025 at 11:57PM John Naylor <johncnaylorls@gmail.com>
wrote:
> >
> > Hmm, clang at -O0 is the same in versions 18, 19, and 20. The change
> > from v17 is easy enough to workaround as in the attached, which
> > produces identical output as master at -O2. Does this fix it for you?
> > If that's not it, it may be more difficult, as the change at -O0 from
> > v16 to v17 is large, so it's hard to see what's going on.
> >
>
> It unfortunately doesn't. I still get the same checksum failure. I tried
> -O0 clang-19, clang-18 and clang-17. I've uploaded the rr recordings
> and initdb output with your patch applied and the debug logs I had.

Okay, in that case we'd better just kludge around it by disabling the
runtime check with clang at -O0 (attached). That will at least get these
kind of builds working while I get more info for a bug report.

--
John Naylor
Amazon Web Services

  6. Re: Invalid control file checksum with AVX-512 during initdb on a clang19 -O0 build

    Andy Fan <zhihuifan1213@163.com> — 2025-06-15T01:23:41Z

    John Naylor <johncnaylorls@gmail.com> writes:

Hi John,

> Okay, in that case we'd better just kludge around it by disabling the
> runtime check with clang at -O0 (attached). That will at least get
> these kind of builds working while I get more info for a bug report.  

I run into the same issue at [1] and verified your patch works. Thanks
for the fix!

[1] https://www.postgresql.org/message-id/87o6uqd3iv.fsf%40163.com

-- 
Best Regards
Andy Fan

  7. Re: Invalid control file checksum with AVX-512 during initdb on a clang19 -O0 build

    John Naylor <johncnaylorls@gmail.com> — 2025-06-16T02:32:54Z

    On Sun, Jun 15, 2025 at 8:23 AM Andy Fan <zhihuifan1213@163.com> wrote:
>
> John Naylor <johncnaylorls@gmail.com> writes:
> > Okay, in that case we'd better just kludge around it by disabling the
> > runtime check with clang at -O0 (attached). That will at least get
> > these kind of builds working while I get more info for a bug report.
>
> I run into the same issue at [1] and verified your patch works. Thanks
> for the fix!

Pushed, thanks for testing, and thanks Soumyadeep for the report!

-- 
John Naylor
Amazon Web Services