Hot snapshot clone fails when baseimage annotation is a bare OCI ref

[tonicmuroq]

Summary

Cloning a hot snapshot can fail when the snapshot manifest's cocoonstack.snapshot.baseimage annotation contains a bare OCI ref such as simular/win10:22h2-20260510.

In our case both the hot snapshot and the referenced base image exist in epoch, but the cocoon node does not have the base qcow2 blob locally. During clone, cocoon tries to auto-pull the bare ref directly and fails with unsupported protocol scheme "", then restore fails because the backing qcow2 file is missing locally.

Environment

• Cocoon cluster: cocoonset-gke, node pool running vk-cocoon
• vm-service env: testing
• Requested hot snapshot: simular/win10-hot-testing:v1-20260511
• Hot snapshot manifest: https://epoch.simular.cloud/v2/simular/win10-hot-testing/manifests/v1-20260511 returns HTTP 200
• Referenced base image manifest: https://epoch.simular.cloud/v2/simular/win10/manifests/22h2-20260510 returns HTTP 200

Hot snapshot manifest annotation:

{
  "cocoonstack.snapshot.baseimage": "simular/win10:22h2-20260510",
  "org.opencontainers.image.created": "2026-05-11T01:42:12Z"
}

Repro

Create/clone a VM from this hot snapshot on a node where the referenced base image is not already present in the local cocoon image cache:

simular/win10-hot-testing:v1-20260511

This was triggered through vm-service's cocoon provider, which created a CocoonSet for a VM named vm-3110f674.

Actual Result

Clone fails during restore:

base image not found locally, pulling simular/win10:22h2-20260510 ...
auto-pull simular/win10:22h2-20260510 failed (imported image?): http get simular/win10:22h2-20260510: Get "simular/win10:22h2-20260510": unsupported protocol scheme "" — clone may fail if base layers are missing
Error: clone VM: vm.restore: PUT http://localhost/api/v1/vm.restore → 500: ["Error from API","The VM could not be restored","Error from device manager","Failed to create QcowDiskAsync","I/O error (path=/var/lib/cocoon/run/cloudhypervisor/4TYF3AE7W2UW7HYC2KNJMC3MBF/overlay.qcow2 op=open)","Backing file I/O error: /var/lib/cocoon/cloudimg/blobs/5c305d620264d10bafd4fb806d3025a146c95a03cffe3372d9a3982229277f55.qcow2","No such file or directory (os error 2)"]

The base image manifest is present in epoch, so this does not appear to be a missing registry tag. It looks like the auto-pull path cannot resolve/pull the bare baseimage ref, and then restore proceeds far enough to require the local backing blob.

Expected Result

One of these should happen:

1. Cocoon resolves bare repo:tag baseimage annotations using the same registry context as the hot snapshot being cloned, or
2. Cocoon requires/stores fully-qualified baseimage annotations and emits a clear validation error before restore, or
3. Clone fails early with an actionable error explaining that the base image must be pre-pulled or fully qualified.

Notes

A fully-qualified base image ref such as epoch.simular.cloud/simular/win10:22h2-20260510 may avoid this, but the current hot snapshot was produced with the bare annotation simular/win10:22h2-20260510.

[CMGS]

Updating with the cocoon + epoch side of the fix and what's left for vk-cocoon.

What landed

• cocoonstack/epoch#4 (deployed): /dl/{name}/{ref} route. ref is either a tag (22h2-20260510, latest) or a manifest digest (sha256:<hex>). Legacy /dl/{name} URLs still work — handler-side fallback retries (name+"/"+ref, latest) and tags the response with Deprecation: true + Link: </dl/{name}/{ref}>; rel="successor-version".
• cocoonstack/cocoon#39: EnsureImage force-pulls when ImageDigest is pinned but missing locally (closes EnsureImage with force=false silently skips re-pull when local URL cache is stale, clone dies in VerifyBaseFiles #37).
• cocoonstack/cocoon#40: EnsureImage validates ref shape against ImageType before pull. Mis-shaped refs short-circuit with an actionable warning ("cloudimg ref X is not an http(s) URL — imported or bare OCI ref?") instead of letting http.Get return the cryptic unsupported protocol scheme \"\". This is the defensive backstop, not the actual fix.

What vk-cocoon still owns

The real fix: write cocoonstack.snapshot.baseimage in the form the base image's own backend speaks. Not all bases are epoch, and OCI-form refs are valid when the base is an OCI image — they just can't leak into the cloudimg channel.

Original VM's ImageType	Base resolved by	Annotation form
cloudimg (epoch /dl/, https://cloud-images.ubuntu.com/..., internal qcow2 mirror, any HTTP source)	cocoon cloudimg backend → plain http.Get	Fully-qualified URL with explicit ref: https://{host}/{path}/{ref}. Host is not assumed to be epoch — any URL with a scheme is fine.
oci (any standard OCI registry: ghcr.io, docker.io, quay.io, private)	cocoon oci backend → name.ParseReference + go-containerregistry	Standard OCI ref: {registry}/{repo}:{tag} or {registry}/{repo}@sha256:{manifest-digest}. No URL scheme — that's how OCI refs are written.

Concretely on the vk-cocoon side, the annotation-writing code should branch on the VM's ImageType:

switch vm.ImageType {
case types.ImageTypeCloudImg:
    // {host} comes from the original pull URL's host, NOT hardcoded to epoch.
    // {ref} is the tag the VM was started with, or the manifest digest if vk-cocoon
    // has it cached from the OCI registry round-trip.
    annotation = fmt.Sprintf("%s/%s/%s", baseURLBase, repo, ref)
case types.ImageTypeOCI:
    annotation = fmt.Sprintf("%s/%s:%s", registryHost, repo, tag)
    // or: fmt.Sprintf("%s/%s@%s", registryHost, repo, manifestDigest)
}

Recommended ref form for each backend

For cloudimg URLs, three forms work post-epoch#4:

Form	Example	When to use
Tag-pinned	https://epoch.simular.cloud/dl/simular/win10/22h2-20260510	Default for hot snapshots — tag is typically timestamped and immutable; cheapest to generate (just string concat at push time)
Manifest-digest-pinned	https://epoch.simular.cloud/dl/simular/win10/sha256:<manifest-digest>	Most reproducible; requires vk-cocoon to grab Docker-Content-Digest header from /v2/{name}/manifests/{tag} at snapshot push time
latest	https://epoch.simular.cloud/dl/simular/win10/latest	Avoid for hot snapshots — reintroduces the issue 37 staleness window

For OCI base images: standard registry conventions, nothing cocoon-specific. name.ParseReference accepts everything go-containerregistry accepts.

One field that's easy to confuse

Don't reuse the same digest in two places:

• The digest in the cloudimg URL (/dl/{name}/sha256:<X>) is the OCI manifest sha256 — epoch resolves this via ManifestJSONByDigest.
• The vmCfg.ImageDigest snapshot field is the blob content sha256 — cocoon stores qcow2 blobs under this name and validates post-pull with Inspect(ImageDigest).

These are different hex values. cocoon's post-pull validation still works regardless of which URL form vk-cocoon chooses, as long as the streamed bytes' sha256 matches vmCfg.ImageDigest.