feat(makefile): host cache + Dockerfile.devrail generator (Story 13.4a)#36
Merged
matthew-on-git merged 1 commit intomainfrom May 4, 2026
Merged
Conversation
First half of Story 13.4 (extended-image build pipeline). Ships the
foundation: host-side persistent plugin cache, Dockerfile.devrail
generator, and a Story 13.3 review-fix fold-in. Story 13.4b will wire
the full docker-build pipeline + DOCKER_RUN swap-in.
Components:
- Makefile: new DEVRAIL_HOST_PLUGINS_CACHE variable
(default ${HOME}/.cache/devrail/plugins) bind-mounted into every
DOCKER_RUN at /opt/devrail/plugins. Plugin manifests fetched by
`make plugins-update` now survive across container invocations.
Closes a Story 13.3 gap (cache was ephemeral).
- Makefile: new _ensure-host-cache target wired as a prereq of every
public host target that invokes DOCKER_RUN. Idempotent mkdir -p.
- scripts/plugin-build-extended-image.sh: reads the loader cache
(Story 13.2 — full manifest content per plugin) and emits a
Dockerfile.devrail extending the core image with each plugin's
container fragment (apt_packages, copy_from_builder, env,
install_script). Output is deterministic — env vars sorted by key,
plugin order matches lockfile order. FROM line pinned to the exact
patch version (not floating :v1) so the eventual devrail-local:<hash>
is stable across local invocations.
- scripts/plugin-resolver.sh: fetch_to_cache now `chmod -R u+rwX,g+rX,
o+rX` the cached tree after the atomic swap. mktemp -d defaults to
0700, which blocked the host user from traversing its own bind-mounted
cache. Closes a Story 13.3 review-fix follow-up.
- tests/test-plugin-build-pipeline.sh: 4-case smoke test:
empty cache → no-op (no Dockerfile written); full container block →
expected dockerfile shape (apt + COPY + ENV sorted + install_script);
deterministic re-runs (sha256 stable); host cache mount + readability
(validates the chmod fold-in end-to-end).
- .github/workflows/ci.yml: new "Plugin build-pipeline smoke test" step.
- CHANGELOG: [Unreleased] Added entry covering all of the above.
Test results (local):
- tests/test-plugin-build-pipeline.sh — 4/4 (new)
- tests/test-plugin-resolver.sh — 16/16 (regression-safe; chmod fold-in
now means Case 16's mtime check could read directly from host, but
kept the sidecar pattern for now to avoid churn)
- tests/test-plugin-loader.sh — 11/11 (regression-safe)
- tests/smoke-rails.sh — 4/4 (regression-safe)
- make _check on dev-toolchain itself — pass
Scope boundary: 13.4a is the foundation. Story 13.4b adds
`_extended-image` build target, `DEVRAIL_RESOLVED_IMAGE` swap-in,
real-build smoke cases (cache hit/miss, multi-plugin layering,
install_script execution, build failure surfacing).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
15 tasks
matthew-on-git
added a commit
that referenced
this pull request
May 4, 2026
* feat(makefile): extended-image build pipeline (Story 13.4b) Completes Story 13.4 (Story 13.4a shipped foundation in PR #36; this PR wires the full build pipeline). When .devrail.yml declares one or more plugins, `make check` (and lint/format/fix/test/security) now auto-builds a project-local extended image (devrail-local:<hash>) that includes core dev-toolchain tools + each plugin's container fragment. Cache hits are free. Components: - scripts/plugin-extended-image.sh: HOST-side orchestrator. Stages install scripts from the host plugin cache into a build-context staging dir (.devrail-plugins-build/), runs `make _generate-dockerfile` inside a container to emit Dockerfile.devrail (which depends on _plugins-load to populate the loader cache), hashes the dockerfile for the tag, checks `docker image inspect` for cache hit, otherwise invokes `docker build`, and writes the resolved tag to .devrail/extended-image-tag. - Makefile: new _generate-dockerfile (in-container) + _extended-image (host) targets. _extended-image wired as a prereq of every public host target that uses DOCKER_RUN. New DEVRAIL_RESOLVED_IMAGE Make variable (recursively-expanded) reads .devrail/extended-image-tag when plugins are declared, else falls back to the core image. DOCKER_RUN switched from := to = expansion to support the swap-in. - tests/fixtures/plugin-repos/minimal-v1/: minimal hermetic fixture with no apt_packages and no copy_from_builder paths — fastest possible build for smoke testing the pipeline end-to-end. - tests/test-plugin-build-pipeline.sh: 9 cases (4 generator unit from 13.4a + 5 full-pipeline). New cases: - Case 5: real docker build → devrail-local:<hash> exists, install script ran (verified via marker file inside the image), env applied. - Case 6: cache hit on second invocation (< 30s end-to-end ceiling). - Case 7: no-plugins regression (no Dockerfile.devrail, no tag file). - Case 8: install_script that exits 1 → structured `error` event with stderr_tail + duration_ms. - Smoke tests bypass `make plugins-update` (which needs a network- reachable git source) by pre-populating the host cache directly and hand-crafting a matching .devrail.lock with the correct content_hash. This isolates the build pipeline as the system under test. Documentation: - CHANGELOG: [Unreleased] Added entry for the full pipeline. - STABILITY: row updated from "Plugin loader + resolver + lockfile" to "Plugin loader + resolver + lockfile + build pipeline" (still Preview; Story 13.5 ships plugin command execution). Test results (local, against freshly built image): - tests/test-plugin-build-pipeline.sh — 9/9 - tests/test-plugin-resolver.sh — 16/16 (regression-safe) - tests/test-plugin-loader.sh — 11/11 (regression-safe) - tests/smoke-rails.sh — 4/4 (regression-safe) - make check on dev-toolchain itself — pass Implementation notes: - DOCKER_RUN swap-in uses recursive expansion (=) so the tag file is re-read each invocation. Without this, immediate evaluation (:=) would capture the tag at make-time before _extended-image had a chance to produce it. - HAS_PLUGINS_DECLARED uses a `yq | awk` pipeline at make-time to detect non-empty plugins:; cheap. - Test fixtures in tests/fixtures/plugin-repos/minimal-v1/ are hermetic (no network, no apt) so the docker-build step is fast enough to run in CI without timeouts. Scope boundary: 13.4 closes here. Story 13.5 (next) implements the plugin command execution loop — the part that actually runs plugin- defined targets during _lint/_format/etc. inside the extended image. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(makefile): address Story 13.4 senior-developer review findings H1: probe `.devrail.yml` parse error vs no-plugins via DEVRAIL_PLUGIN_PROBE, so a malformed config fails Make loudly instead of silently skipping the extended-image build. H2: source the host orchestrator from the dev-toolchain image when consumers inherit only the Makefile (no `scripts/` dir locally) — extracted to `.devrail/host-bin/`, cached and invalidated by image tag. M1+L6: flock per-workspace `.devrail/.build.lock` so concurrent `make check` invocations on the same checkout don't race on STAGING_DIR or Dockerfile.devrail. M2: source `lib/plugin-cache.sh` in the orchestrator and use the shared `derive_slug` helper instead of duplicating the basename/.git logic. M3: surface `make plugins-update` as the actionable hint when the host cache is empty. L1: document host-side requirements (yq v4+, sha256sum, flock, docker buildx) in STABILITY.md. L2: WORKSPACE override via DEVRAIL_WORKSPACE env for testability. L3: BUILD_LOG cleanup folded into the EXIT trap. L4: forward DEVRAIL_QUIET / DEVRAIL_DEBUG to the in-container generator for consistent log behaviour. L5: explicit `docker buildx version` precondition check with a clear error. Tests: - L7: two-plugin smoke (Case 10) — exercises the for-loop over plugin entries. - L8: tighten Case 6 cache-hit ceiling from 30s to 10s (closer to the 1s AC). - L9: Case 8 now asserts no tag file is written on build failure. - L10: Case 9 transition test — removing plugins clears the stale tag file. - L11: Case 11 end-to-end — plugins-update + _extended-image against a file:// fixture, exercising the full resolver → loader → build path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(makefile): defer H1 yq-parse error from Make-time to _extended-image The previous H1 fix used `$(error ...)` at Make parse time, which fired before `make _plugins-update` could invoke the resolver script — regressing the resolver-test Case 12 that expects the resolver to surface the "config could not be parsed by yq" event. Move the error to runtime inside `_extended-image` instead. Other targets (`_plugins-update`, `_check`) still parse Make successfully and let their in-container scripts handle malformed YAML with their existing structured events. `_extended-image` itself emits the same event format and exits 2. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(makefile): clear stale extended-image tag on plugins-removed transition The `if [ -n "$(HAS_PLUGINS_DECLARED)" ]` gate skipped the orchestrator entirely when plugins were removed from `.devrail.yml`, so the orchestrator's tag-file cleanup never ran. DOCKER_RUN would keep referencing a phantom `devrail-local:<hash>` tag from a prior build. Add Makefile-level cleanup in the elif arm of `_extended-image`. Also gate `_devrail-host-bin` extraction on HAS_PLUGINS_DECLARED so consumer repos that don't declare plugins skip the docker-cp roundtrip on every `make check`. Caught by the L10 transition test (Case 9) added in the prior commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(makefile): match Case 9 transition assertion to Makefile-emitted event The transition cleanup now happens at the Makefile level (orchestrator is not invoked when plugins are removed). Assert the Makefile's "plugins removed; clearing stale extended-image tag" event instead of the orchestrator's "no plugins declared; using core image" message. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(makefile): match Case 10 plugin section regex to generator format The generator emits `# --- plugin: <name>@<rev> (source: <url>) ---`, not `# plugin:`. Update the count regex anchor accordingly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(makefile): mount WORKDIR for Case 11 file:// resolution Case 11 used `make plugins-update` from the host, which goes through the standard DOCKER_RUN — that doesn't mount the fixture path inside the container, so the resolver can't reach `file:///tmp/...`. Mirror Case 4's harness: invoke `_plugins-update` directly via docker run with $WORKDIR bind-mounted at the same path inside the container. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First half of Story 13.4 (extended-image build pipeline). Ships the foundation: host-side persistent plugin cache, Dockerfile.devrail generator, and a Story 13.3 review-fix fold-in. Story 13.4b (next PR) will wire the full docker-build pipeline + DOCKER_RUN swap-in.
Components
Makefile`scripts/plugin-build-extended-image.sh`
Reads the Story 13.2 loader cache (full manifest content per plugin merged with resolution metadata) and emits a workspace-local `Dockerfile.devrail` extending the core image with each plugin's `container:` fragment (`apt_packages`, `copy_from_builder`, `env`, `install_script`).
scripts/plugin-resolver.sh— review-fix fold-in`fetch_to_cache` now `chmod -R u+rwX,g+rX,o+rX` the cached tree after the atomic swap. `mktemp -d` defaults to 0700, which blocked the host user from traversing its own bind-mounted cache (Story 13.3 Case 16 hit this and worked around it via a sidecar docker container; this fix would let that workaround go away — kept for now to avoid churn).
Tests
`tests/test-plugin-build-pipeline.sh` — 4-case smoke:
CI + Docs
Acceptance criteria from Story 13.4
This PR addresses AC 1, AC 6, AC 7 (partial — generator emits the install_script COPY/RUN; full execution is verified in 13.4b).
Story 13.4b will address AC 2 (build pipeline), AC 3 (cache hit), AC 4 (DOCKER_RUN swap-in), AC 5 (no-plugins regression at the build pipeline level), AC 8 (full smoke coverage).
Test results (local)
Behaviour change for consumers
No `plugins:` in `.devrail.yml`: zero change. Host cache dir gets created (`mkdir -p`) but stays empty.
Has `plugins:`: `make plugins-update` cache survives container exits. `Dockerfile.devrail` is NOT yet auto-generated by `make check` (that's Story 13.4b); generator must be invoked manually for now.
🤖 Generated with Claude Code