[ML] Automate version bump in CI pipeline by edsavage · Pull Request #3018 · elastic/ml-cpp

edsavage · 2026-04-07T03:51:13Z

Summary

Replaces the manual block step in the version-bump pipeline with automated version bump logic, supporting both patch and minor release workflows.

Patch workflow (`WORKFLOW=patch`, default)

Bump version — checks out $BRANCH, updates elasticsearchVersion in gradle.properties to $NEW_VERSION, commits as elasticsearchmachine, pushes
Fetch DRA Artifacts — polls staging + snapshot artifact URLs until the new version is available

Minor workflow (`WORKFLOW=minor`, feature freeze day)

Create minor branch — derives the branch name from the current version on $BRANCH (e.g., main at 9.4.0 → branch 9.4), pushes it (inherits the current version)
Bump upstream — bumps $BRANCH (e.g., main) to $NEW_VERSION (e.g., 9.5.0), commits, pushes
Fetch DRA Artifacts — two parallel steps:
- Upstream branch: polls for $NEW_VERSION staging + snapshot
- Minor branch: polls for the inherited version staging + snapshot

Pattern

Follows the established Elasticsearch repo pattern for automated commits from CI:

elasticsearchmachine / infra-root+elasticsearchmachine@elastic.co as committer
Local git config (not --global)
git diff-index --quiet HEAD for idempotency
git pull --ff-only before push to handle concurrent commits

Script features

DRY_RUN=true — performs all steps except git push
Portable sed -i (macOS/Linux)
Idempotent: skips if version already matches, skips branch creation if it already exists
Reusable helpers: git_push, sed_inplace, configure_git, bump_version_on_branch
Unknown WORKFLOW values rejected with clear error
Exports MINOR_BRANCH/MINOR_VERSION via Buildkite meta-data for downstream steps

Pipeline features

WORKFLOW env var selects pipeline shape at generation time
Minor branch name/version derived from NEW_VERSION (e.g., 9.5.0 → 9.4 branch, 9.4.0 version)
DRA step generation refactored into helper functions

Files

dev-tools/bump_version.sh — standalone script with patch + minor support
.buildkite/job-version-bump.json.py — pipeline generator with workflow-aware step generation

Test plan

prodsecmachine · 2026-04-07T03:51:27Z

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

edsavage · 2026-04-10T03:58:12Z

valeriy42 · 2026-04-10T10:00:02Z

@edsavage Just to get context: is this needed for the new automated version-bump orchestration strategy?

edsavage · 2026-04-12T22:50:24Z

@edsavage Just to get context: is this needed for the new automated version-bump orchestration strategy?

Yes, ultimately. I'm just tinkering with a few things at the moment. I'll make it a draft for now.

Replace the manual block step in the version-bump pipeline with an automated step that: 1. Checks out the target branch 2. Updates elasticsearchVersion in gradle.properties to $NEW_VERSION 3. Commits as elasticsearchmachine 4. Pushes directly to the branch (no PR needed) Follows the same pattern as Elasticsearch's automated Lucene snapshot updates (.buildkite/scripts/lucene-snapshot/update-es-snapshot.sh). The Fetch DRA Artifacts step now depends on the bump step, ensuring the version is updated before polling for artifacts at the new version. Made-with: Cursor

Adds a DRY_RUN=true option that performs all steps (checkout, sed, commit) but skips the final git push. Useful for testing the pipeline and for local verification. Also makes sed portable across macOS/Linux and uses local git config instead of --global. Made-with: Cursor

Extends the version bump script and pipeline to support the minor release workflow (feature freeze day) in addition to patch bumps. Script changes (bump_version.sh): - Accepts WORKFLOW env var: 'patch' (default) or 'minor' - Minor workflow: creates a new minor branch from BRANCH (e.g., 9.4 from main inheriting current version), then bumps BRANCH to NEW_VERSION (the next minor) - Exports MINOR_BRANCH and MINOR_VERSION via Buildkite meta-data for downstream steps - Idempotent: skips branch creation if it already exists on remote - Refactored into reusable helpers (git_push, sed_inplace, configure_git, bump_version_on_branch) Pipeline changes (job-version-bump.json.py): - Reads WORKFLOW env var at pipeline generation time - Patch: 1 bump step + 1 DRA step (2 artifact checks) — unchanged - Minor: 1 bump step + 2 parallel DRA steps (upstream + minor branch, 4 artifact checks total covering staging + snapshot for both branches) - Minor branch name/version derived from NEW_VERSION at generation time (e.g., NEW_VERSION=9.5.0 → minor branch 9.4, version 9.4.0) - Refactored DRA step generation into helper functions Tested locally with DRY_RUN=true on a throwaway branch: - Minor workflow correctly creates branch, bumps upstream - Patch workflow backward-compatible - Unknown workflow rejected with clear error Made-with: Cursor

The top-level notify block in dynamically uploaded pipelines doesn't fire reliably for build-level state changes. Replace it with the same two-stage pattern used by the other ml-cpp pipelines: a step that uploads send_slack_notification.sh as a sub-pipeline, which has its own notify block that fires on build completion. Made-with: Cursor

When bumping the version on main, the script now also updates .backportrc.json so the new version label maps to main. This prevents the backport action from trying to backport v9.X.0 PRs to a non-existent 9.X branch. Uses Python for reliable cross-platform JSON manipulation rather than sed on a JSON file. Made-with: Cursor

edsavage · 2026-04-30T04:42:32Z

@copilot resolve the merge conflicts in this pull request

Copilot

Pull request overview

Automates the ML CI version-bump pipeline by replacing the manual approval block with a scripted git-based bump (patch/minor workflows) and workflow-aware DRA artifact polling.

Changes:

Add dev-tools/bump_version.sh to implement patch/minor version bump automation (commit/push + optional minor-branch creation).
Refactor .buildkite/job-version-bump.json.py to generate different pipeline shapes based on WORKFLOW and to run the bump script before artifact polling.
Factor out DRA polling step construction into helper functions.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
`dev-tools/bump_version.sh`	New automation script to bump `gradle.properties` (and update backport config on `main`) for patch/minor workflows.
`.buildkite/job-version-bump.json.py`	Pipeline generator updates to run the bump step and then poll DRA artifacts (patch vs minor).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-30T22:15:04Z

+mapping = data.get('branchLabelMapping', {})
+new_mapping = {}
+for k, v in mapping.items():
+    if v == 'main' and re.match(r'\^v\d+\.\d+\.\d+\\\$', k):


The .backportrc.json update logic won’t match keys like ^v9.4.0$ because the regex expects a literal backslash before the trailing $ (re.match(r'\^v\d+\.\d+\.\d+\\\$', k)). As a result, the branchLabelMapping entry pointing to main will never be updated to the new version. Adjust the regex to match the actual key format in .backportrc.json (e.g., a full match for ^v<semver>$).

Suggested change

if v == 'main' and re.match(r'\^v\d+\.\d+\.\d+\\\$', k):

if v == 'main' and re.fullmatch(r'\^v\d+\.\d+\.\d+\$', k):

Copilot · 2026-04-30T22:15:04Z

+        local escaped_version
+        escaped_version=$(echo "$target_version" | sed 's/\./\\./g')


escaped_version is computed but never used. This makes the script harder to maintain and can hide real issues in shellcheck/linting; please remove it or use it for the intended replacement logic.

Suggested change

local escaped_version

escaped_version=$(echo "$target_version" | sed 's/\./\\./g')

Copilot · 2026-04-30T22:15:05Z

+            "label": "Queue a :slack: notification for the pipeline",
+            "depends_on": None,
+            "command": ".buildkite/pipelines/send_slack_notification.sh | buildkite-agent pipeline upload",
+            "agents": {


This step pipes .buildkite/pipelines/send_slack_notification.sh into buildkite-agent pipeline upload, but that script currently emits a malformed Slack channel string (“#machine-learn-build" with a curly quote + mismatched quote). That will likely prevent notifications from being delivered (or could break pipeline upload if the YAML is parsed strictly). Fix the channel quoting in the notification script or switch to emitting the notify block directly from this generator.

edsavage marked this pull request as draft April 12, 2026 22:50

edsavage and others added 4 commits April 13, 2026 10:54

edsavage force-pushed the feature/version-bump-automation branch from 3804708 to ffe046c Compare April 12, 2026 22:54

edsavage added the v9.5.0 label Apr 13, 2026

edsavage requested a review from Copilot April 30, 2026 22:10

Copilot started reviewing on behalf of edsavage April 30, 2026 22:11 View session

Copilot AI reviewed Apr 30, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Automate version bump in CI pipeline#3018

[ML] Automate version bump in CI pipeline#3018
edsavage wants to merge 5 commits intoelastic:mainfrom
edsavage:feature/version-bump-automation

edsavage commented Apr 7, 2026 •

edited

Loading

Uh oh!

prodsecmachine commented Apr 7, 2026 •

edited

Loading

Uh oh!

edsavage commented Apr 10, 2026 •

edited

Loading

Uh oh!

valeriy42 commented Apr 10, 2026

Uh oh!

edsavage commented Apr 12, 2026

Uh oh!

edsavage commented Apr 30, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 30, 2026

Uh oh!

Copilot AI Apr 30, 2026

Uh oh!

Copilot AI Apr 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	if v == 'main' and re.match(r'\^v\d+\.\d+\.\d+\\\$', k):
	if v == 'main' and re.fullmatch(r'\^v\d+\.\d+\.\d+\$', k):

		local escaped_version
		escaped_version=$(echo "$target_version" \| sed 's/\./\\./g')

Conversation

edsavage commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Patch workflow (WORKFLOW=patch, default)

Minor workflow (WORKFLOW=minor, feature freeze day)

Pattern

Script features

Pipeline features

Files

Test plan

Uh oh!

prodsecmachine commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Snyk checks have passed. No issues have been found so far.

Uh oh!

edsavage commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Status checklist

Core functionality

Pipeline / DRA artifact verification

Branch protection

Testing

Spec items not yet addressed

Not applicable to ml-cpp

Uh oh!

valeriy42 commented Apr 10, 2026

Uh oh!

edsavage commented Apr 12, 2026

Uh oh!

edsavage commented Apr 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

edsavage commented Apr 7, 2026 •

edited

Loading

Patch workflow (`WORKFLOW=patch`, default)

Minor workflow (`WORKFLOW=minor`, feature freeze day)

prodsecmachine commented Apr 7, 2026 •

edited

Loading

edsavage commented Apr 10, 2026 •

edited

Loading