Skip to content

feat(catalog): runtime sync engine + metadata public/private repo split#223

Merged
github-actions[bot] merged 5 commits intomainfrom
claude/dreamy-williams-d8b6f9
Apr 26, 2026
Merged

feat(catalog): runtime sync engine + metadata public/private repo split#223
github-actions[bot] merged 5 commits intomainfrom
claude/dreamy-williams-d8b6f9

Conversation

@typelicious
Copy link
Copy Markdown
Collaborator

@typelicious typelicious commented Apr 26, 2026

Summary

  • Splits fusionAIze/fusionaize-metadata into a new public companion (fusionaize-metadata-public) so anonymous installs can pull pricing/model data, while routing heuristics + operator quotas stay private.
  • Lands a runtime catalog sync engine in faigate (HTTPS + ETag conditional GET) that resolves through private → public → bundled so installed instances pick up new models without a faigate release.
  • Bumps catalog schema to v1.1 (additive: tier_status, expires_at) and ships first content updates for claude-opus-4-7, claude-haiku-4-5, gpt-5.5-codex reasoning tiers, and DeepSeek V4 backing.

What's included

New modules

  • faigate/metadata_catalog_sync.py — HTTPS fetcher with conditional GET, structural payload validation, secrets-not-logged invariant
  • faigate/catalog_cache.py — ~/.cache/faigate/metadata/{tier}/ filesystem cache, atomic temp+rename, best-effort fcntl lock
  • faigate/catalog_resolver.py — private → public → bundled resolution chain, TTL-gated cache, force_refresh, status() telemetry
  • faigate/models_cli.py — new faigate-models CLI: update {--check, --diff}, status [--json]
  • faigate/assets/metadata/catalog.v1.json (+ __init__.py) — bundled snapshot loaded via importlib.resources; closes the offline / first-boot fallback gap (faigate previously had no fallback in the wheel)
  • scripts/refresh-bundled-catalog — pre-release snapshot regen

Wire-in (minimal)

  • provider_catalog.py:_load_external_catalog falls through to CatalogResolver when no local file is found. Existing FAIGATE_PROVIDER_METADATA_FILE/DIR env vars still win for dev workflows.

Tests

  • 25 new tests in tests/test_metadata_catalog_sync.py (HTTP status codes, ETag handling, schema validation, fallback chain, cache roundtrip, secrets scrubber).
  • All 49 catalog-related tests pass (25 new + 24 existing).

Docs

  • docs/CATALOG-UPDATER.md — auth setup (fine-grained PAT), env vars, CLI usage, troubleshooting
  • docs/FUSIONAIZE-SHARED-METADATA.md — updated for the public/private split with new repo layouts and the dual env-var pattern
  • docs/blueprints/model-updater/{prd.md,prd.json,progress.txt,agents.md} — full blueprint for traceability

Companion changes (already merged in metadata repos)

  • fusionaize-metadata-public afe5801 — initial public catalog, then schema v1.1 + new models on main
  • fusionaize-metadata 95ab0ab — moved public-eligible content out, retained products/, packages/; CI guards added so it survives without providers/catalog.v1.json

Test plan

  • pytest tests/test_metadata_catalog_sync.py — 25/25 pass
  • pytest tests/test_provider_catalog{,_refresh}.py tests/test_dashboard_provider_catalog.py — 24/24 pass (no regressions)
  • Live end-to-end: faigate-models update against raw.githubusercontent.com — 41 providers, ETag captured, cache populated
  • faigate-models status reports both remote and bundled tiers correctly
  • Bundled snapshot loadable via importlib.resources (verified by status output bundled present=yes providers=41)
  • Reviewer: confirm pricing values for the new entries should be flagged as freshness_status: stale (they are — no real prices are claimed; this PR ships the structural updates only)
  • Reviewer: spot-check that FAIGATE_PROVIDER_METADATA_FILE override still wins over the resolver chain (preserves dev workflow)

Deferred to follow-ups

These are explicitly out of scope so this PR stays reviewable. All tracked in docs/blueprints/model-updater/prd.json:

  • CODE-007 — 24h daemon tick (today: lazy refresh on first miss + manual CLI)
  • CODE-008 — wire sync alerts through existing build_catalog_alerts
  • HOT-002 — OpenRouter pricing audit (needs live research)
  • Pricing verification for the new model entries (all currently freshness_status: stale)
  • Roadmap: cosign/GPG signing of catalog payloads

🤖 Generated with Claude Code

André Lange and others added 2 commits April 26, 2026 14:36
@claude
docs(metadata): split fusionaize-metadata into public + private repos
1269e75 
Phase 2 of the model-updater blueprint:

Public repo created: github.com/fusionAIze/fusionaize-metadata-public
  Holds providers/, models/, offerings/, schemas (catalog data, anonymous
  read access). Source SHA migrated from fusionaize-metadata@2c271b8.

Private repo retained: github.com/fusionAIze/fusionaize-metadata
  Holds products/gate/overlays.v1.json + packages/catalog.v1.json
  (routing heuristics + operator quotas — IP-sensitive).

This commit ships:
  - docs/FUSIONAIZE-SHARED-METADATA.md updated with the split,
    new repo layouts, and the dual env var pattern for dev environments
  - docs/blueprints/model-updater/{prd.md,prd.json,progress.txt,agents.md}
    — full blueprint for the model-updater work (HOT patches, repo split,
    sync engine, schema v1.1, content fill, tests, docs; ~7.5 dev days,
    21 atomic Ralph-Loop tasks + 4 roadmap items)

No faigate code changes yet. Phase 3 (MetadataCatalogSync, daemon tick,
CLI, schema bump) is the next milestone — see prd.json HOT-001 → DOCS-001.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
@claude
feat(catalog): runtime sync engine with HTTPS+ETag, CLI, bundled fall…
4d03e92 
…back

Implements Phase 3 of the model-updater blueprint: faigate now pulls the
curated provider catalog from a remote repo at runtime so installed
instances pick up new models and pricing without a faigate release.

New modules:
  - faigate/metadata_catalog_sync.py: HTTPS fetcher with conditional GET
    (If-None-Match), redacted token logging, structural payload validation
  - faigate/catalog_cache.py: per-tier filesystem cache (~/.cache/faigate/
    metadata/{public,private}/) with atomic temp+rename writes and a
    best-effort fcntl lock for CLI/daemon coordination
  - faigate/catalog_resolver.py: private→public→bundled chain, TTL-gated
    cache, force_refresh, status() telemetry. ResolverConfig.from_env()
    reads FAIGATE_METADATA_TOKEN/PUBLIC_URL/PRIVATE_URL/REFRESH_INTERVAL_SECONDS.
  - faigate/models_cli.py: faigate-models {update, status} commands with
    --check, --diff, --json flags. Registered in pyproject.toml.
  - faigate/assets/metadata/catalog.v1.json: bundled snapshot loaded via
    importlib.resources for offline / first-boot fallback.
  - scripts/refresh-bundled-catalog: regenerate the bundled snapshot
    pre-release.

Wire-in:
  - provider_catalog.py:_load_external_catalog falls through to
    CatalogResolver when no local file is present. FAIGATE_PROVIDER_METADATA_*
    env vars still win for dev workflows.

Tests: 25 new tests in test_metadata_catalog_sync.py covering HTTP status
codes, ETag conditional GET, schema validation, fallback chain, cache
roundtrip, and the secrets-not-logged invariant. All 49 catalog-related
tests (new + existing) pass.

Live verification: faigate-models update and status commands fetched
the public catalog from raw.githubusercontent.com successfully (41
providers, ETag captured, bundled snapshot detected).

Deferred to follow-up:
  - 24h daemon tick (CODE-007)
  - alert integration via build_catalog_alerts (CODE-008)
  - HOT-002 openrouter pricing audit (needs live research)
  - cosign/GPG signing of catalog payloads (roadmap)

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
@github-actions github-actions Bot enabled auto-merge (squash) April 26, 2026 18:03
André Lange and others added 3 commits April 26, 2026 20:06
@claude
chore: fix ruff lint findings (unused imports, Iterator location)
420b87a 
Auto-applied ruff --fix:
- catalog_cache.py: import Iterator from collections.abc, not typing
- catalog_resolver.py: drop unused CachedCatalog and FetchResult imports
- models_cli.py: drop unused 'time' and DEFAULT_REFRESH_INTERVAL_SECONDS
- test_metadata_catalog_sync.py: drop unused time/FetchResult/HttpFetcher,
  fix import sort order

No behavior change. 49/49 catalog tests still pass.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
@claude
chore: apply ruff format to new catalog modules
f58a9d5 
CI lint job runs both `ruff check` and `ruff format --check`. The previous
commit only fixed check findings; this collapses the wrapped lines that
format wanted on one line each. No behavior change.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
@claude
fix(catalog): trailing newline + relax network-error test prefix asse…
3da0c6e 
…rtion

Two CI fixes:
- Add trailing newline to faigate/assets/metadata/catalog.v1.json so
  the pre-commit end-of-file-fixer hook is satisfied.
- test_fetch_network_error_returns_error_does_not_raise: don't assert
  on the "network:" error prefix. httpx 0.27/0.28 drift across CI
  Python versions means httpx.ConnectError sometimes hits the generic
  Exception branch instead of the httpx.HTTPError branch. The contract
  the test cares about — error status returned, no exception bubbled —
  still holds; assert on the message body instead.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
@github-actions github-actions Bot merged commit f04b981 into main Apr 26, 2026
18 checks passed
@typelicious typelicious mentioned this pull request Apr 27, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@typelicious