Thread

Re: Improving tracking/processing of buildfarm test failures

Alexander Lakhin <exclusion@gmail.com> — 2025-11-01T08:00:00Z

Hello hackers,

Please take a look at the October report on buildfarm failures:
# SELECT br, COUNT(*) FROM failures WHERE dt >= '2025-10-01' AND
  dt < '2025-11-01' GROUP BY br;
REL_13_STABLE: 18
REL_14_STABLE: 9
REL_15_STABLE: 4
REL_16_STABLE: 8
REL_17_STABLE: 13
REL_18_STABLE: 66
master: 64
-- Total: 182
(Counting test failures only, excluding indent-check, Configure, Build
errors.)

# SELECT COUNT(*) FROM (SELECT DISTINCT issue_link FROM failures WHERE
  dt >= '2025-10-01' AND dt < '2025-11-01');
25

# SELECT issue_link, COUNT(*) FROM failures WHERE dt >= '2025-10-01' AND
  dt < '2025-11-01' GROUP BY issue_link ORDER BY 2 DESC LIMIT 6;
https://www.postgresql.org/message-id/CAMbWs4_VxjdM-nBvt8YE%3D84rE4OLBES27Wz1P0%3D9Z6KgwPqzEA%40mail.gmail.com : 41
-- Fixed

https://www.postgresql.org/message-id/2874644f-6431-41f4-abe2-99e5ab052606%40gmail.com : 28
-- Hurd

https://www.postgresql.org/message-id/2e327a21-9954-4d05-aa9e-3b7fe926f39f%40joeconway.com : 16
-- Fixed

https://www.postgresql.org/message-id/aPAl27_urHSODwRN%40paquier.xyz : 16
-- Fixed

https://www.postgresql.org/message-id/66541690-91c2-48e6-9c99-12c9af25012b%40joeconway.com : 10


# SELECT COUNT(*) FROM failures WHERE dt >= '2025-10-01' AND
  dt < '2025-11-01' AND issue_link IS NULL; -- Unsorted/unhelpful failures
16

Short-lived failures:
75

Best regards,
Alexander