Resolve AppKit and Agent Skills versions from compatibility manifest

[pkosiec]

Summary

Introduces a CLI compatibility manifest (internal/build/cli-compat.json) that maps CLI versions to compatible AppKit template and Agent Skills versions. This enables template updates to reach users without CLI releases.

Manifest format

The manifest is purely range-based — each versioned entry defines a range floor that applies to that CLI version and all versions above it, up to the next entry. The manifest should be sparse: only add a new entry when a compatibility boundary changes (e.g., new AppKit templates require specific CLI features).

Resolution

• Exact match → use that entry
• Between entries → nearest lower version
• Newer than all → highest versioned entry
• Dev builds (0.0.0-dev*) → highest versioned entry

Manifest sources (fallback chain)

1. Fresh local cache (< 1h)
2. Remote fetch from GitHub (with retry)
3. Stale local cache or embedded manifest fallback

Set DATABRICKS_FORCE_EMBEDDED_COMPAT=true to skip remote fetch and use only the embedded manifest (useful for local development).

Companion PRs

• databricks/appkit#333
• databricks/databricks-agent-skills#64

Screenshot

[renaudhartert-db]

Thanks for the PR @pkosiec. Could we have a chat internally about what you're trying to achieve? I'd like to make sure that this is aligned with the overall direction we're planning to evolve that command toward.

[simonfaltum]

The new libs/clicompat/ package rolls its own file cache (readLocalManifest, writeLocalManifest, manual DATABRICKS_CACHE_DIR / DATABRICKS_CACHE_ENABLED handling, hand-written temp-file-then-rename, mtime-based TTL). The CLI already has a shared cache library for this in libs/cache, and it's what every other on-disk cache in the repo uses. Let's not introduce a second caching convention.

Use libs/cache

• cache.NewCache(ctx, component, ttl, metrics) + cache.GetOrCompute[T] / cache.Get[T] / cache.Put[T] handle JSON serialization, fingerprinting, atomic writes, per-key locking, env-var controls, and metrics integration.
• Storage path: ~/.cache/databricks/<cli-version>/<component>/<hash>.json. Version-scoped, swept by databricks cache clear.
• Introduced in Initial implementation of the local cache layer #3678.

Closest analog: the .well-known/databricks-config cache (#5011)

Same shape as this PR: remote HTTP fetch with a TTL and fallback behavior. See libs/hostmetadata/resolver.go, ~90 lines total. It wraps the SDK's HostMetadataResolver with cache.GetOrCompute for the hit-or-fetch path and cache.Get for a read-only negative-cache probe. Positive TTL 1h, negative TTL 60s.

Other call sites worth skimming

• bundle/config/mutator/populate_current_user.go (Enable caching user identity by default #4202) caches CurrentUser.Me(). Shows GetOrCompute with a real fingerprint.
• bundle/config/mutator/initialize_cache.go shows the metrics wiring at bundle init.

Sketch for clicompat

const (
    compatCacheComponent = "cli-compat"
    compatCacheTTL       = 1 * time.Hour
)

type manifestFingerprint struct{}

func FetchManifest(ctx context.Context) (Manifest, error) {
    c := cache.NewCache(ctx, compatCacheComponent, compatCacheTTL, nil)
    m, err := cache.GetOrCompute[Manifest](ctx, c, manifestFingerprint{}, fetchRemoteWithRetry)
    if err == nil {
        return m, nil
    }
    return parseEmbeddedManifest()
}

That removes cachedManifest, isFresh, readLocalManifest, writeLocalManifest, manifestLocalPath, and the manual env-var handling (~100 lines).

One open question: the stale-cache fallback (current tier 3a)

libs/cache doesn't expose a "return the cached value even if expired" read. Two options:

1. Drop tier 3a and rely on the embedded manifest. The embedded copy is always in the binary, so we lose only the narrow window of "local cache just expired and remote happens to be down right now." Seems like a fair trade for adopting the shared primitive.
2. Add cache.Peek / cache.GetStale to libs/cache as a follow-up, then preserve the 4-tier behavior.

I'd vote (1) for this PR.

[pkosiec]

@simonfaltum yeah, as discussed, it was done on purpose to fallback to a successfuly fetched manifest (even if it is "outdated") in case of GitHub is down 👍

[simonfaltum]

Did a deep dive on this in person, looks good

[renaudhartert-db]

LGTM on the high-level, deferring final approval to Simon.