Thread
Commits
-
ci: Improve OpenBSD core dump backtrace handling.
- b498af4204bd 19 (unreleased) landed
- 5114d62e7499 18.1 landed
-
ci: Improve OpenBSD core dump backtrace handling
Nazir Bilal Yavuz <byavuz81@gmail.com> — 2025-10-24T13:20:00Z
Hi, While working on [1], I realized that the core dump output of OpenBSD has very little information. We had an off-list chat with Thomas and found that this happens because OpenBSD core dump files do not include the executable information; so we need to run `lldb` with the executable path. Attached patch manually finds executables in the build directory and calls `lldb` with these executables' paths'. Here is an example: Master [2]: ``` [11:36:51.769] (lldb) target create --core "/var/crash/postgres.core" [11:36:51.797] Core file '/var/crash/postgres.core' (x86_64) was loaded. [11:36:51.797] (lldb) thread backtrace all [11:36:51.798] * thread #1, stop reason = signal SIGABRT [11:36:51.798] * frame #0: 0x0000029d071e14bb [11:36:51.798] frame #1: 0x550327f2b69964ee [11:36:51.798] (lldb) quit ``` Patched [3]: ``` [11:22:11.087] (lldb) target create "/home/postgres/postgres/build/tmp_install/usr/local/pgsql/bin/postgres" --core "/var/crash/postgres.core" [11:22:13.104] Core file '/var/crash/postgres.core' (x86_64) was loaded. [11:22:13.104] (lldb) thread backtrace all [11:22:13.105] warning: This version of LLDB has no plugin for the language "assembler". Inspection of frame variables will be limited. [11:22:13.213] * thread #1, stop reason = signal SIGABRT [11:22:13.213] * frame #0: 0x00000b12e4dfe27b libc.so.100.3`_thread_sys_thrkill at -:2 [11:22:13.213] frame #1: 0x00000b12e4db010b libc.so.100.3`_libc_abort at abort.c:51:8 [11:22:13.213] frame #2: 0x00000b126d3fd39d pg_prewarm.so`pg_prewarm(fcinfo=<unavailable>) at pg_prewarm.c:80:2 [11:22:13.213] frame #3: 0x00000b0ff584add5 postgres`ExecInterpExpr(state=0x00000b12527e4790, econtext=0x00000b12527e4438, isnull=0x0000000000000000) at execExprInterp.c:926:8 [11:22:13.213] frame #4: 0x00000b0ff5896a47 postgres`ExecResult [inlined] ExecEvalExprNoReturn(state=0x00000b12527e4790, econtext=0x00000b12527e4438) at executor.h:420:13 [11:22:13.213] frame #5: 0x00000b0ff5896a3b postgres`ExecResult [inlined] ExecEvalExprNoReturnSwitchContext(state=0x00000b12527e4790, econtext=0x00000b12527e4438) at executor.h:461:2 [11:22:13.213] frame #6: 0x00000b0ff5896a2a postgres`ExecResult [inlined] ExecProject(projInfo=0x00000b12527e4788) at executor.h:493:2 [11:22:13.213] frame #7: 0x00000b0ff5896a11 postgres`ExecResult(pstate=<unavailable>) at nodeResult.c:135:10 [11:22:13.213] frame #8: 0x00000b0ff5855eda postgres`standard_ExecutorRun [inlined] ExecProcNode(node=0x00000b12527e4328) at executor.h:316:9 [11:22:13.213] frame #9: 0x00000b0ff5855ec5 postgres`standard_ExecutorRun [inlined] ExecutePlan(queryDesc=0x00000b124f332500, operation=CMD_SELECT, sendTuples=<unavailable>, numberTuples=0, direction=<unavailable>, dest=0x00000b12e5cae7a0) at execMain.c:1697:10 [11:22:13.213] frame #10: 0x00000b0ff5855e4a postgres`standard_ExecutorRun(queryDesc=0x00000b124f332500, direction=<unavailable>, count=0) at execMain.c:366:3 [11:22:13.213] frame #11: 0x00000b0ff5a7623b postgres`PortalRunSelect(portal=0x00000b128d07c100, forward=<unavailable>, count=0, dest=<unavailable>) at pquery.c:921:4 [11:22:13.213] frame #12: 0x00000b0ff5a75d9d postgres`PortalRun(portal=0x00000b128d07c100, count=9223372036854775807, isTopLevel=<unavailable>, dest=0x00000b12e5cae7a0, altdest=0x00000b12e5cae7a0, qc=0x0000759da39141b0) at pquery.c:765:18 [11:22:13.213] frame #13: 0x00000b0ff5a74ad9 postgres`exec_simple_query(query_string="SELECT pg_prewarm('test', 'read');") at postgres.c:1279:10 [11:22:13.213] frame #14: 0x00000b0ff5a72332 postgres`PostgresMain(dbname=<unavailable>, username=<unavailable>) at postgres.c:0 [11:22:13.213] frame #15: 0x00000b0ff5a6bfad postgres`BackendMain(startup_data=<unavailable>, startup_data_len=<unavailable>) at backend_startup.c:124:2 [11:22:13.213] frame #16: 0x00000b0ff59a35a6 postgres`postmaster_child_launch(child_type=B_BACKEND, child_slot=1, startup_data=0x0000759da3915030, startup_data_len=24, client_sock=0x0000759da3914f20) at launch_backend.c:268:3 [11:22:13.213] frame #17: 0x00000b0ff59a8584 postgres`ServerLoop [inlined] BackendStartup(client_sock=<unavailable>) at postmaster.c:3598:8 [11:22:13.213] frame #18: 0x00000b0ff59a8543 postgres`ServerLoop at postmaster.c:1713:6 [11:22:13.213] frame #19: 0x00000b0ff59a60e8 postgres`PostmasterMain(argc=<unavailable>, argv=<unavailable>) at postmaster.c:1403:11 [11:22:13.213] frame #20: 0x00000b0ff58c6915 postgres`main(argc=4, argv=0x0000759da39155c8) at main.c:231:4 [11:22:13.213] frame #21: 0x00000b0ff565728b postgres`__start + 299 ``` Any feedback would be appreciated. [1] https://postgr.es/m/CAN55FZ12R-LG=wZZ8yLY7+tmVbR=9Y_O5uveDHU0nmYT+AiTjQ@mail.gmail.com [2] https://cirrus-ci.com/task/6177664672727040 [3] https://cirrus-ci.com/task/5487875443130368 -- Regards, Nazir Bilal Yavuz Microsoft -
Re: ci: Improve OpenBSD core dump backtrace handling
Thomas Munro <thomas.munro@gmail.com> — 2025-10-24T21:41:17Z
On Sat, Oct 25, 2025 at 2:20 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: > Any feedback would be appreciated. + filename=$(basename "$corefile") + base=$(echo "$filename" | sed 's/\.core.*$//') + binary=$(find "$executable_directory" -type f -name "$base" 2>/dev/null | head -n 1) + + if [ -z "$binary" ]; then + echo "${base} executable can not be found in ${executable_directory}" + continue + fi + + lldb "$binary" -c "$corefile" --batch -o 'thread backtrace all' -o 'quit' s/can not/cannot/, I don't make the rules... Maybe if not found it should just run it the old way without the executable? We don't really expect other programs to be dumped (well, sometimes we abort cp etc, something I plan to fix...), and they're probably stripped anyway, but I guess we might as well try to show as much information as we can if it happens? Would it be better to set PATH to $executable_directory:$PATH and use "which"? -
Re: ci: Improve OpenBSD core dump backtrace handling
Nazir Bilal Yavuz <byavuz81@gmail.com> — 2025-10-30T11:58:48Z
Hi, Thank you for looking into this! On Sat, 25 Oct 2025 at 00:41, Thomas Munro <thomas.munro@gmail.com> wrote: > > On Sat, Oct 25, 2025 at 2:20 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: > > Any feedback would be appreciated. > > + filename=$(basename "$corefile") > + base=$(echo "$filename" | sed 's/\.core.*$//') > + binary=$(find "$executable_directory" -type f -name "$base" > 2>/dev/null | head -n 1) > + > + if [ -z "$binary" ]; then > + echo "${base} executable can not be found in > ${executable_directory}" > + continue > + fi > + > + lldb "$binary" -c "$corefile" --batch -o 'thread backtrace > all' -o 'quit' > > s/can not/cannot/, I don't make the rules... > > Maybe if not found it should just run it the old way without the > executable? We don't really expect other programs to be dumped (well, > sometimes we abort cp etc, something I plan to fix...), and they're > probably stripped anyway, but I guess we might as well try to show as > much information as we can if it happens? Would it be better to set > PATH to $executable_directory:$PATH and use "which"? I agree with all of your points. All of them are addressed in the v2. -- Regards, Nazir Bilal Yavuz Microsoft -
Re: ci: Improve OpenBSD core dump backtrace handling
Thomas Munro <thomas.munro@gmail.com> — 2025-11-02T23:52:04Z
On Fri, Oct 31, 2025 at 12:59 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: > I agree with all of your points. All of them are addressed in the v2. LGTM, so I'll go and commit this shortly. Thanks!