Skip to content

Reduce test-suite CI disk usage with profile.ci and runner cleanup#141

Open
cderv wants to merge 7 commits intomainfrom
ci-disk-space-profile-ci
Open

Reduce test-suite CI disk usage with profile.ci and runner cleanup#141
cderv wants to merge 7 commits intomainfrom
ci-disk-space-profile-ci

Conversation

@cderv
Copy link
Copy Markdown
Member

@cderv cderv commented Apr 29, 2026

The test-suite workflow on ubuntu-latest hit "No space left on device" during cargo nextest run, even with the existing free-disk-space step already reclaiming 18.8 GB. The workspace has grown past PR #55's mitigation.

Root Cause

Two compounding pressures:

  1. The default [profile.dev] emits full debuginfo (debug = 2), which roughly doubles target/ size on a workspace this big. Dependencies dominate target/ and carry no useful debug info for triaging q2 failures.

  2. The workflow had both cargo build and cargo nextest run steps. Library crates compiled with --cfg test have a different fingerprint than dev builds and produce separate .rlib artifacts in target/<profile>/. With ~35 crates that runs into multi-GB duplication at peak disk.

Fix

Four independent levers, applied in one workflow change:

  • Add [profile.ci] to the workspace Cargo.toml (inherits from dev, debug = "line-tables-only" on the workspace, debug = false on all dependencies). Local builds keep full debuginfo via the unchanged dev profile.
  • Replace cargo build + cargo nextest run with a single cargo nextest run --tests --cargo-profile ci. Per the nextest design docs, nextest delegates compilation to cargo test --no-run, which makes the separate cargo build step redundant: the lib + bins + integration-tests that --tests selects strictly supersede cargo build's default targets (lib + bins, non-test). --tests rather than --all-targets because quarto-yaml's harness = false benches print prose reports that nextest can't enumerate as tests; the prior workflow never built benches anyway, so coverage matches. RUSTFLAGS=-D warnings continues to apply (it's a rustc env var, picked up regardless of which cargo subcommand drives the build).
  • Enable remove_tool_cache: true on the existing endersonmenezes/free-disk-space step (~6 GB more). Verified safe: pre-installed rustup data lives in /etc/skel/.{rustup,cargo} and dtolnay/rust-toolchain installs to ~/.rustup/toolchains/, neither of which is in /opt/hostedtoolcache/.
  • Add docker image prune --all --force + docker builder prune -a --force (~3-8 GB more). Pre-pulled docker images aren't used by this job.

Full rationale, measured numbers, and held-in-reserve options (swap removal, df -h diagnostic, larger runner) are in claude-notes/2026-04-28-ci-disk-space-and-profile-ci.md.

cderv added 6 commits April 29, 2026 09:51
CI run 25062065055 (PR #139, ubuntu-latest) hit "No space left on
device" during `cargo nextest run` even though
`endersonmenezes/free-disk-space` had already freed 18.8 GB. The
workspace has grown past PR #55's mitigation.

Add a design doc capturing the situation, the two-part fix being
applied next (a [profile.ci] Cargo profile + dropping the redundant
`cargo build` step in CI), what's lost in CI vs locally, and held-in-
reserve options if more headroom is needed.

The Cargo.toml comment and workflow comment in the follow-up commits
point here for the full rationale.
Inherits everything from the dev profile (including the
`opt-level = "s"` wasm-bindgen workaround) but strips most debuginfo:
`debug = "line-tables-only"` on the workspace and `debug = false` on
all dependencies.

The default `dev` profile is unchanged, so local builds keep full
debuginfo. The new profile is opt-in via `--cargo-profile ci`, and
will be activated in the test-suite workflow in the next commit.

Background and measured rationale:
claude-notes/2026-04-28-ci-disk-space-and-profile-ci.md
Replace the standalone `cargo build` step + `cargo nextest run` step
with a single `cargo nextest run --all-targets --cargo-profile ci`.

Per the nextest design docs, `cargo nextest run` invokes the
equivalent of `cargo test --no-run` to compile, and `cargo test`'s
default target selection is `--lib --bins --tests`. Adding
`--all-targets` makes this a strict superset of `cargo build
--all-targets`, so removing the separate build step does not skip
any compilation. `RUSTFLAGS=-D warnings` is a rustc env var passed
to every rustc invocation, so warning enforcement is unchanged.

The redundancy that disappears: `cargo build` and `cargo nextest run`
share `target/<profile>/` but library crates compiled with `--cfg
test` have a different fingerprint and produce separate `.rlib`
artifacts alongside the dev-build ones. With ~35 crates that
duplication ran into multi-GB at peak disk.

`--cargo-profile ci` activates the profile added in the previous
commit, stripping debuginfo for further `target/` reduction.

Background and measured rationale:
claude-notes/2026-04-28-ci-disk-space-and-profile-ci.md
Add `remove_tool_cache: true` to the existing endersonmenezes/free-
disk-space step (~6 GB) and prune Docker images and build cache at
the start of the job (~3-8 GB).

`tool_cache: true` removes /opt/hostedtoolcache/ which holds Node,
Python, Go, Ruby, .NET preinstalled binaries. Safe here because the
test-suite job uses none of those — Rust comes from rustup
(dtolnay/rust-toolchain), pandoc from dpkg, tree-sitter from apt+curl,
nextest/insta from taiki-e/install-action's own cache.

Docker prune is independent of the free-disk-space action. ubuntu-
latest images include several pre-pulled containers that no step in
this job uses.

Updates the explainer doc table to reflect that levers C and D moved
from "held in reserve" to "done"; only swap_storage, df -h diagnostic,
and a larger runner remain in reserve.
`--all-targets` enumerates bench targets, and quarto-yaml's
`memory_overhead` / `scaling_overhead` benches are declared
`harness = false`. Nextest can't parse their prose output as a
libtest listing and fails with "creating test list failed".

`--tests` keeps the consolidation goal — one nextest-driven build
covering lib (test + non-test rlibs), bins, and integration tests,
so the separate `cargo build` step stays gone — while matching
prior CI coverage exactly (plain `cargo build` never built benches).

Doc note updated to cite cargo-build's `--tests` definition (whose
"lib may be built twice" clause is the load-bearing fact for the
no-duplicate-rlib argument) and to record why `--all-targets` was
the wrong flag.
The integration test hardcoded `target/debug/q2`, which broke under
`--cargo-profile ci` (binary lands at `target/ci/q2`). Derive the
profile directory from the test binary's own location instead, so
the test works under any profile without env-var coupling.

`CARGO_BIN_EXE_<name>` would have been the standard fix, but it's
package-scoped — the LSP test lives in `quarto-lsp` while the bin
is defined in `quarto`, so cargo never sets it here.

claude-note updated with rationale and the assert_cmd alternative
(deferred until a test actually wants its assertion API).
@cderv
Copy link
Copy Markdown
Member Author

cderv commented Apr 29, 2026

Follow-up fix: quarto-lsp integration test was hardcoding target/debug/q2, which broke under --cargo-profile ci (binary now lands at target/ci/q2).

Approach chosen: derive the profile directory from std::env::current_exe() — the test binary itself sits at target/<profile>/deps/, so backing up two parents lands in the same target/<profile>/ where the production bin is built. Profile-correct on all platforms with no env coupling.

Why not CARGO_BIN_EXE_<name>: that env var is package-scoped — Cargo only sets it for integration tests in the same package as the bin. Our LSP test lives in quarto-lsp while the bin is in quarto, so it's never set here.

Why not assert_cmd: assert_cmd::cargo::cargo_bin("q2") does exactly the same thing internally and is the idiomatic crate-based answer. We deferred adding it because this test doesn't use any of assert_cmd's actual value — no .assert(), no stdout matchers, just JSON-RPC over stdio. The trigger to switch is the first test that wants cargo run -- style command-driving with output assertions; at that point cargo_bin() is the standard binary-discovery helper that comes with the crate.

Full rationale in claude-notes/2026-04-28-ci-disk-space-and-profile-ci.md (Change 3 section).

The cleanup-levers table cell for "Drop redundant cargo build" still
referenced `--all-targets`, which we tried first but replaced with
`--tests` to skip `quarto-yaml`'s harness=false benches. Update the
cell to match the workflow. Also tighten the TL;DR wording on Docker
prune ordering — it runs right after the free-disk-space step, not
"at the start of the job".
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant