Horton

Re: Non-reproducible AIO failure

Tom Lane <tgl@sss.pgh.pa.us>

View thread
From: Tom Lane <tgl@sss.pgh.pa.us>
To: Thomas Munro <thomas.munro@gmail.com>
Cc: Alexander Lakhin <exclusion@gmail.com>, Andres Freund <andres@anarazel.de>, pgsql-hackers@lists.postgresql.org
Date: 2025-05-26T00:25:46Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. aio: Stop using enum bitfields due to bad code generation

  2. amcheck: Fix posting tree checks in gin_index_check()

  3. aio: Add missing memory barrier when waiting for IO handle

Thomas Munro <thomas.munro@gmail.com> writes:
> On Sun, May 25, 2025 at 3:22 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> So far, I've failed to get anything useful out of core files
>> from this failure.  The trace goes back no further than
>> (lldb) bt
>> * thread #1
>> * frame #0: 0x000000018de39388 libsystem_kernel.dylib`__pthread_kill + 8

> (And Alexander reported the same off-list.). It's interesting that the
> elog.c backtrace stuff is able to analyse the stack and it looks
> normal AFAICS.  Could that be interfering with the stack in the core?!

No, but something is.  Just to make sure it wasn't totally broken,
I added a sure-to-fail Assert in a random place (I chose
pg_backend_pid), and I get both a trace in the postmaster log and a
perfectly usable core file:

TRAP: failed Assert("MyProcPid == 0"), File: "pgstatfuncs.c", Line: 692, PID: 59063
0   postgres                            0x00000001031f1fa4 ExceptionalCondition + 108
1   postgres                            0x00000001031672b4 pg_stat_get_backend_pid + 0
2   postgres                            0x0000000102e9e598 ExecInterpExpr + 5524
3   postgres                            0x0000000102edb100 ExecResult + 368
4   postgres                            0x0000000102ea6418 standard_ExecutorRun + 316

(lldb) bt
* thread #1
  * frame #0: 0x00000001836b5388 libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x00000001836ee88c libsystem_pthread.dylib`pthread_kill + 296
    frame #2: 0x00000001835f7c60 libsystem_c.dylib`abort + 124
    frame #3: 0x000000010491dfac postgres`ExceptionalCondition(conditionName=<unavailable>, fileName=<unavailable>, lineNumber=692) at assert.c:66:2 [opt]
    frame #4: 0x000000010489329c postgres`pg_backend_pid(fcinfo=<unavailable>) at pgstatfuncs.c:692:2 [opt]
    frame #5: 0x00000001045ca598 postgres`ExecInterpExpr(state=0x000000013780d190, econtext=0x000000013780ce38, isnull=<unavailable>) at execExprInterp.c:0 [opt]
    frame #6: 0x0000000104607100 postgres`ExecResult [inlined] ExecEvalExprNoReturn(state=<unavailable>, econtext=0x000000013780ce38) at executor.h:417:13 [opt]
    frame #7: 0x00000001046070f4 postgres`ExecResult [inlined] ExecEvalExprNoReturnSwitchContext(state=<unavailable>, econtext=0x000000013780ce38) at executor.h:458:2 [opt]

The fact that I can trace through this Assert failure but not the
AIO one strongly suggests some system-level problem in the latter.
There is something rotten in the state of Denmark.

For completeness, this is with Sequoia 15.5 (latest macOS) on
an M4 Pro MacBook.

> but I haven't seen this failure on my little M4 MacBook Air yet
> (Sequoia 15.5, Apple clang-1700.0.13.3).  It is infected with
> corporate security-ware that intercepts at least file system stuff and
> slows it down and I can't even convince it to dump core files right
> now.

As far as that goes, if you have SIP turned on (which I'm sure a
corporate laptop would), there are extra steps needed to get a
core to happen.  See [1]; that page is old, but the recipe still
works for me.

			regards, tom lane

[1] https://nasa.github.io/trick/howto_guides/How-to-dump-core-file-on-MacOS.html