[ML] Add AI-powered build failure analysis to CI pipelines

[edsavage]

Summary

• Adds a new Buildkite pipeline step that automatically analyses failed builds using the Anthropic Claude API
• When a build fails, the step fetches logs from failed steps and posts a structured diagnosis (root cause, classification, suggested fix, confidence) as a Buildkite annotation
• The step is soft-fail and only runs when the build is actually failing (if: "build.state == 'failed' || build.state == 'failing'")
• Wired into all three pipelines: PR builds, nightly snapshot builds, and nightly debug builds
• Claude API key stored in Vault at secret/ci/elastic-ml-cpp/anthropic/claude

New files

• dev-tools/analyze_build_failure.py — core analysis script
• .buildkite/pipelines/analyze_build_failure.yml.sh — pipeline step definition

Test plan

• Tested locally with --dry-run against real failed builds (snapshot #5819, debug [7.8][ML] Add a constant to the prediction which minimises the unregularised loss for classification and regression #1194)
• Verify Vault secret is accessible from CI agents
• Trigger a PR build and confirm the step appears (it should be skipped if the build passes)

Made with Cursor

[prodsecmachine]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[valeriy42]

Will this create a GitHub comment on the PR with fix suggestions?

[edsavage]

Will this create a GitHub comment on the PR with fix suggestions?

No, just as an annotation on the Buildkite build. I guess it could be a GH comment as well though, which would notify the user and have better visibility if changes are suggested.

[edsavage]

Pushed a new commit that adds GitHub PR comments for build failure analysis. When the build is a PR build, the analysis is now posted as a comment directly on the PR (in addition to the Buildkite annotation and optional Slack notification).

Key details:

• Uses BUILDKITE_PULL_REQUEST env var (set automatically by Buildkite) to detect PR builds
• Posts via GitHub API using a token from Vault (secret/ci/elastic-ml-cpp/github/pr_comment_token)
• Uses an HTML comment marker (<!-- build-failure-analysis -->) to find and update existing comments on rebuild/retry, avoiding duplicates
• Non-PR builds (nightly snapshots, etc.) continue to use Buildkite annotations only

This addresses @valeriy42's feedback about improving visibility for PR authors.

[valeriy42]

Thank you for adding GH comment functionality. I think it makes a lot of sense to reduce friction so this information is visible to the developer.

My last concern is about burning the API tokens. Is it possible to activate this function per GitHub comment?

I think we have the following user story:
As a developer, I want to ask the CI system why it failed and what needs to be done to fix it.

[edsavage]

User Experience: Examining a Failed Build

After this PR merges, here's what a developer sees when a PR build fails:

1. Immediate feedback — GitHub commit statuses (existing)

Red/green marks appear on the PR for each Buildkite step (e.g. "Build on Linux x86_64 RelWithDebInfo — failed"). Each is a clickable link to the Buildkite step. This is the existing publish_commit_status_per_step behaviour.

2. Native PR comment from @elasticmachine (new)

Once the build completes, a PR comment appears listing what failed:

💔 Build Failed

• Buildkite Build (link)

Failed CI Steps

• Build on Linux x86_64 RelWithDebInfo (link)
• Test on Linux x86_64 RelWithDebInfo (link)

History

• 💛 Build [ML] Platform independent dependency report script #2350 was flaky (previous commit)

This is automatic (no opt-in), and includes build history across commits on the PR, flagging flaky builds. Enabled by ELASTIC_PR_COMMENTS_ENABLED in catalog-info.yaml.

3. AI analysis (opt-in for PR builds)

If the failure isn't obvious, the developer comments on the PR:

buildkite analyze

This triggers the analyze_build_failure step which fetches logs from failed steps, sends them to Claude for diagnosis, and posts a Buildkite annotation at the top of the build page.

4. AI analysis PR comment from @github-actions[bot] (new)

A GitHub Actions workflow picks up the analyze step's commit status and posts a PR comment:

🔍 Build Failure Analysis

Root Cause

The CMultiFileDataAdderTest test failed due to a temp file collision...

Classification

test failure

Suggested Fix

Use process ID for unique temp file names...

Confidence

high — The error message clearly indicates...

This comment is updated in-place on subsequent analyses (not duplicated). No personal access token or GitHub App is required — the workflow uses the built-in GITHUB_TOKEN.

5. Slack + email (nightly/snapshot builds)

For non-PR builds, the analysis is also posted to #machine-learn-build via Slack. Email notifications to build-machine-learning@elastic.co continue as before.

---

Summary

When	Where	What	Source
Per-step	PR checks	Red/green status with Buildkite links	Buildkite (existing)
Build complete	PR comment	Failed steps + build history	@elasticmachine (new)
After buildkite analyze	Buildkite build page	AI diagnosis annotation	Buildkite annotation
After buildkite analyze	PR comment	AI diagnosis with root cause + fix	@github-actions[bot] (new)

The developer can stop at any layer depending on how obvious the failure is. Most of the time steps 1–2 are enough; the AI analysis is there for non-obvious cases.

[edsavage]

buildkite analyze

[edsavage]

buildkite analyze

[edsavage]

buildkite test this

[edsavage]

buildkite run_qa_tests on linux, macos, windows aarch64, x86_64

[edsavage]

buildkite run_qa_tests on linux, macos, windows x86_64, aarch64

[edsavage]

buildkite test this

[edsavage]

buildkite run_qa_tests

[Copilot]

Pull request overview

Introduces an AI-based Buildkite failure-analysis capability that fetches logs from failed jobs, generates a structured diagnosis via Anthropic Claude, and publishes results to Buildkite annotations and (intended) GitHub PR comments.

Changes:

• Add dev-tools/analyze_build_failure.py to extract error context from Buildkite job logs and request an LLM diagnosis (plus optional Slack posting).
• Add a Buildkite pipeline step generator .buildkite/pipelines/analyze_build_failure.yml.sh and wire it into nightly pipelines plus a new buildkite analyze PR-comment action.
• Add a GitHub Actions workflow to post/update PR comments by fetching analysis from Buildkite build metadata.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
dev-tools/analyze_build_failure.py	New failure analysis script: Buildkite API log fetch, excerpting, Claude call, Buildkite annotation + metadata, optional Slack
.buildkite/pipelines/analyze_build_failure.yml.sh	New Buildkite step definition for running the analysis script and depending on build steps
.github/workflows/post-build-analysis.yml	New workflow to read Buildkite metadata and post/update a PR comment
.buildkite/pipeline.json.py	Adds “analyze-only” lightweight pipeline path for PR comment trigger
.buildkite/ml_pipeline/config.py	Adds run_analyze flag parsed from PR comment action
.buildkite/pull-requests.json	Extends trigger comment regex to allow buildkite analyze action
.buildkite/job-build-test-all-debug.json.py	Wires analysis step into nightly debug pipeline and computes build step keys
.buildkite/branch.json.py	Wires analysis step into nightly snapshot pipeline
.buildkite/hooks/post-checkout	Fetches Buildkite read token + Anthropic key + Slack webhook from Vault for the analysis step
catalog-info.yaml	Adds an environment flag intended to enable PR comments in the PR build pipeline configuration

Comments suppressed due to low confidence (1)

.buildkite/pipeline.json.py:65

• PR description says the failure-analysis step is wired into PR builds, but in the normal PR build path this file never appends the analyze-build-failure step (it’s only added in the run_analyze lightweight pipeline). Either add the step to the standard PR pipeline (like nightly pipelines do) or update the PR description/comments to match the actual behavior.

    pipeline_steps.append(pipeline_steps.generate_step("Queue a :slack: notification for the pipeline",
                                                       ".buildkite/pipelines/send_slack_notification.sh"))
    pipeline_steps.append(pipeline_steps.generate_step("Queue a :email: notification for the pipeline",
                                                       ".buildkite/pipelines/send_email_notification.sh"))
    pipeline_steps.append(pipeline_steps.generate_step("Upload clang-format validation",
                                                       ".buildkite/pipelines/format_and_validation.yml.sh"))

    # Compute which build step keys will exist so that analytics and
    # failure-analysis steps can emit a correct depends_on list.
    build_step_keys = []
    if config.build_linux and config.build_aarch64:
        build_step_keys.append("build_test_linux-aarch64-RelWithDebInfo")
    if config.build_linux and config.build_x86_64:
        build_step_keys.append("build_test_linux-x86_64-RelWithDebInfo")
    if config.build_macos and config.build_aarch64:
        build_step_keys.append("build_test_macos-aarch64-RelWithDebInfo")
    if config.build_windows and config.build_x86_64:
        build_step_keys.append("build_test_Windows-x86_64-RelWithDebInfo")

    env = {
        "VERSION_QUALIFIER": "",
        "ML_BUILD_STEP_KEYS": ",".join(build_step_keys),
    }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The step definition doesn’t include the conditional mentioned in the PR description (only run when the build is failing). As written, the analysis step will run on successful builds too (and require secrets / call Buildkite APIs), which is unnecessary and could incur external API cost. Add an if: guard on the step (e.g., build.state == 'failed' || build.state == 'failing') or an equivalent gating mechanism.

[Copilot]

buildkite-agent meta-data set is invoked without providing a value argument. The Buildkite agent CLI expects a value (or a --value/--file flag); stdin isn’t used for the value, so this will fail or store an empty value and the GitHub Actions workflow won’t be able to fetch the analysis. Pass the metadata value explicitly (prefer --file to avoid command-length/escaping issues).

[Copilot]

The script exits if ANTHROPIC_API_KEY is missing before it checks whether the target build actually failed. This means the step still requires the Claude key even when the build passed (or when no failed jobs are found), which defeats the “only run on failure” intent and makes local/dry-run usage harder. Move the Claude key requirement to just before calling Claude (after confirming there are failed jobs to analyze).

[Copilot]

Failed jobs are filtered to state == 'failed' only. Buildkite can mark failing jobs as timed_out (and potentially other terminal states), so the script may print “No failed steps found” even when the build is failing. Include additional failure states (at least timed_out) when selecting jobs to analyze.

Suggested change

	failed_jobs = [
	j for j in build.get("jobs", [])
	if j.get("type") == "script" and j.get("state") == "failed"
	failed_job_states = {"failed", "timed_out"}
	failed_jobs = [
	j for j in build.get("jobs", [])
	if j.get("type") == "script" and j.get("state") in failed_job_states

[Copilot]

The script forwards (parts of) raw Buildkite logs to an external LLM API. Currently only ANSI/timestamp markers are removed; there’s no redaction for secrets (tokens, API keys, credentials, URLs with embedded auth, etc.). Add a redaction pass before sending log_excerpt to Claude (and before posting to Slack/annotations), and consider explicitly disabling analysis when the build is handling sensitive secrets.

[Copilot]

The here-doc that builds BODY is indented, which will inject leading spaces into the posted comment content. In GitHub-flavored Markdown, leading spaces can turn the entire comment into a code block and break formatting. Remove the extra indentation inside the here-doc (or use a <<-EOF heredoc with tabs) so the marker/header start at column 0.

Suggested change

	${MARKER}
	## :mag: Build Failure Analysis
	${ANALYSIS}
	---
	[View Buildkite build](${BUILD_URL}) | *Analysis generated by Claude. Verify before acting.*
	EOF
	${MARKER}
	## :mag: Build Failure Analysis
	${ANALYSIS}
	---
	[View Buildkite build](${BUILD_URL}) | *Analysis generated by Claude. Verify before acting.*
	EOF

[Copilot]

depends_on is built only from ML_BUILD_STEP_KEYS. In pipelines that have separate test steps (e.g. branch.json.py sets ML_TEST_STEP_KEYS), this allows the analysis step to start before tests finish and it may miss test failures entirely. Consider including test step keys in depends_on (or using a combined env var) so the analysis runs after all relevant jobs have completed.

Suggested change

	# Emit depends_on dynamically — ML_BUILD_STEP_KEYS is a comma-separated
	# list of step keys set by the pipeline generator. In analyze-previous
	# mode there are no build steps so this block is skipped.
	if [ -n "${ML_BUILD_STEP_KEYS:-}" ]; then
	echo ' depends_on:'
	IFS=',' read -ra STEP_KEYS <<< "$ML_BUILD_STEP_KEYS"
	for key in "${STEP_KEYS[@]}"; do
	# Emit depends_on dynamically — ML_BUILD_STEP_KEYS and ML_TEST_STEP_KEYS
	# are comma-separated lists of step keys set by the pipeline generator.
	# In analyze-previous mode there may be no build or test steps, so this
	# block is skipped when neither list is present.
	DEPENDS_ON_KEYS=()
	if [ -n "${ML_BUILD_STEP_KEYS:-}" ]; then
	IFS=',' read -ra STEP_KEYS <<< "$ML_BUILD_STEP_KEYS"
	DEPENDS_ON_KEYS+=("${STEP_KEYS[@]}")
	fi
	if [ -n "${ML_TEST_STEP_KEYS:-}" ]; then
	IFS=',' read -ra STEP_KEYS <<< "$ML_TEST_STEP_KEYS"
	DEPENDS_ON_KEYS+=("${STEP_KEYS[@]}")
	fi
	if [ "${#DEPENDS_ON_KEYS[@]}" -gt 0 ]; then
	echo ' depends_on:'
	for key in "${DEPENDS_ON_KEYS[@]}"; do

[Copilot]

Buildkite API calls don’t set explicit timeouts and have limited error handling. buildkite_get() has no timeout and get_job_log() only catches HTTPError, so transient network/DNS issues can crash the script and prevent annotations/metadata from being published. Add reasonable timeouts and catch URLError/timeouts around Buildkite API and log fetches, returning a clean “could not fetch” analysis instead of raising.

[Copilot]

The Buildkite meta-data endpoint returns JSON (e.g. { key, value }). This workflow currently writes the entire JSON response into analysis.md, so the PR comment will contain JSON rather than the actual analysis text. Parse the response and extract .value (e.g. with jq -r) before writing to the file.

Suggested change

	ANALYSIS=$(curl -sS -f \
	-H "Authorization: Bearer ${BK_TOKEN}" \
	"https://api.buildkite.com/v2/organizations/elastic/pipelines/${PIPELINE}/builds/${BUILD_NUM}/meta-data/build-failure-analysis" \
	2>/dev/null) || true
	if [ -z "$ANALYSIS" ]; then
	echo "No analysis metadata found — skipping."
	echo "skip=true" >> "$GITHUB_OUTPUT"
	exit 0
	fi
	# Save to file to avoid shell quoting issues.
	echo "$ANALYSIS" > /tmp/analysis.md
	ANALYSIS_JSON=$(curl -sS -f \
	-H "Authorization: Bearer ${BK_TOKEN}" \
	"https://api.buildkite.com/v2/organizations/elastic/pipelines/${PIPELINE}/builds/${BUILD_NUM}/meta-data/build-failure-analysis" \
	2>/dev/null) || true
	if [ -z "$ANALYSIS_JSON" ]; then
	echo "No analysis metadata found — skipping."
	echo "skip=true" >> "$GITHUB_OUTPUT"
	exit 0
	fi
	ANALYSIS=$(printf '%s' "$ANALYSIS_JSON" | jq -r '.value // empty')
	if [ -z "$ANALYSIS" ]; then
	echo "Analysis metadata did not contain a value — skipping."
	echo "skip=true" >> "$GITHUB_OUTPUT"
	exit 0
	fi
	# Save to file to avoid shell quoting issues.
	printf '%s\n' "$ANALYSIS" > /tmp/analysis.md

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)