diff --git a/docs/docs/development/templates.md b/docs/docs/development/templates.md index 50cbeba2e..028f95b3f 100644 --- a/docs/docs/development/templates.md +++ b/docs/docs/development/templates.md @@ -50,9 +50,14 @@ The plugin manifest drives the CLI's behavior during `databricks apps init`: - **`app.yaml` generation** — resource fields produce `env` + `valueFrom` entries - **`databricks.yml` generation** — resource fields produce bundle variables and app resource entries +The synced manifest is generated by `appkit plugin sync --write` from each plugin's `manifest.json` (see [Plugin manifest](../plugins/manifest.md) for the authoring contract). The on-disk shape carries a `version` field that the CLI uses to negotiate features: + +- `"1.0"` / `"1.1"` — earlier shapes; still readable. +- `"2.0"` — current shape. Adds `scaffolding` (required) and the `origin` field on every resource field entry. JSON Schema published at `https://databricks.github.io/appkit/schemas/template-plugins.schema.json`. + ### Resource field properties -Each resource field in the manifest can have these properties: +Each resource field in the synced manifest can have these properties: | Property | Description | |----------|-------------| @@ -63,6 +68,21 @@ Each resource field in the manifest can have these properties: | `value` | Default value used when no user input is provided | | `resolve` | Auto-populated by CLI from API calls instead of prompting (see below) | | `examples` | Example values shown in field descriptions | +| `discovery` | How the CLI lists candidate values for the field (see [Plugin manifest — Resource discovery](../plugins/manifest.md#resource-discovery)). | +| `origin` | **v2.0** computed field. How the value is determined — see below. | + +### `origin` (v2.0) + +`origin` is computed at sync time from the field's other properties — plugin authors do not write it. It tells scaffolding agents how each value reaches the running app: + +| Origin | Trigger | Meaning | +|--------|---------|---------| +| `"platform"` | `localOnly: true` | Auto-injected by Databricks Apps at deploy time. Generated for local `.env` only; absent from `app.yaml` and bundle variables. | +| `"static"` | `value` set | Hardcoded literal. CLI does not prompt. | +| `"cli"` | `resolve` set | Resolved by the CLI from API calls (e.g. `postgres:host`). | +| `"user"` | none of the above | User must provide the value at init time. | + +Precedence is in the order above (`localOnly` wins over `value`, which wins over `resolve`). The transform overwrites any hand-edited `origin` on the next sync — drift between the on-disk value and the field's actual shape is not possible by construction. ### Resolvers @@ -89,7 +109,64 @@ Example field definition: } ``` +After sync, the field carries `"origin": "platform"` (because `localOnly` takes precedence over `resolve` for local-only fields injected at deploy time). + +### `postScaffold` propagation + +Each plugin's `postScaffold` array is propagated unchanged into its entry in `appkit.plugins.json`. The CLI surfaces these instructions to the user after init runs. See [Post-scaffold steps](../plugins/manifest.md#post-scaffold-steps) for the field shape. + +### `scaffolding` descriptor (v2.0) + +The `scaffolding` block at the top level of `appkit.plugins.json` describes the scaffolding command and the rules a scaffolding agent must follow. It is required when `version` is `"2.0"`. + +```json +{ + "scaffolding": { + "command": "databricks apps init", + "flags": { + "--template-dir": { + "description": "Path to the template directory containing the app scaffold", + "required": true + }, + "--config-dir": { + "description": "Path to the output directory for the initialized app", + "required": true + }, + "--profile": { + "description": "Databricks CLI profile to use for authentication", + "required": false + } + }, + "rules": { + "never": [ + "Modify files inside the template directory", + "Skip resource configuration prompts", + "Hardcode workspace-specific values in template files" + ], + "must": [ + "Use the template manifest (appkit.plugins.json) as the source of truth for available plugins", + "Respect requiredByTemplate flags when presenting plugin selection", + "Generate .env files with all required environment variables from selected plugins", + "When discovering volume resources, prompt the user for catalog and schema before listing volumes." + ] + } + } +} +``` + +| Field | Description | +|-------|-------------| +| `command` | Scaffolding command the agent should invoke. | +| `flags` | Map of flag name to `{ description, required?, pattern?, default? }`. | +| `rules.never` | Actions the scaffolding agent must never perform. | +| `rules.must` | Actions the scaffolding agent must always perform. | + +Each rule item is capped at **120 characters** by the schema. Long prose fails validation — split into discrete actionable directives. + +The descriptor is canonical: AppKit owns the values and ships them with every synced template manifest. Authors of consuming agents (LLM-driven scaffolders, custom CLI runners) should treat the rule lists as enforcement contracts, not suggestions. + ## See also -- [Plugin management](../plugins/plugin-management.md) — `appkit plugin sync`, `appkit plugin create` -- [Configuration](../configuration.mdx) — environment variables +- [Plugin manifest](../plugins/manifest.md) — the authoring side (`manifest.json`). +- [Plugin management](../plugins/plugin-management.md) — `appkit plugin sync`, `appkit plugin create`. +- [Configuration](../configuration.mdx) — environment variables. diff --git a/docs/docs/plugins/custom-plugins.md b/docs/docs/plugins/custom-plugins.md index 297ecde09..3c905a6bf 100644 --- a/docs/docs/plugins/custom-plugins.md +++ b/docs/docs/plugins/custom-plugins.md @@ -18,34 +18,41 @@ For a deeper understanding of the plugin structure, read on. ## Basic plugin example -Extend the [`Plugin`](../api/appkit/Class.Plugin.md) class and export with `toPlugin()`: +Author the manifest as JSON, import it, and attach it to a [`Plugin`](../api/appkit/Class.Plugin.md) subclass via `static manifest`. Export with `toPlugin()`: + +```json +// my-plugin/manifest.json +{ + "$schema": "https://databricks.github.io/appkit/schemas/plugin-manifest.schema.json", + "name": "my-plugin", + "displayName": "My Plugin", + "description": "A custom plugin", + "resources": { + "required": [ + { + "type": "secret", + "alias": "apiKey", + "resourceKey": "api-key", + "description": "API key for external service", + "permission": "READ", + "fields": { + "scope": { "env": "MY_SECRET_SCOPE", "description": "Secret scope" }, + "key": { "env": "MY_API_KEY", "description": "Secret key name" } + } + } + ], + "optional": [] + } +} +``` ```typescript +// my-plugin/index.ts import { Plugin, toPlugin, type PluginManifest } from "@databricks/appkit"; -import type express from "express"; +import manifest from "./manifest.json"; class MyPlugin extends Plugin { - static manifest = { - name: "myPlugin", - displayName: "My Plugin", - description: "A custom plugin", - resources: { - required: [ - { - type: "secret", - alias: "apiKey", - resourceKey: "apiKey", - description: "API key for external service", - permission: "READ", - fields: { - scope: { env: "MY_SECRET_SCOPE", description: "Secret scope" }, - key: { env: "MY_API_KEY", description: "Secret key name" } - } - } - ], - optional: [] - } - } satisfies PluginManifest<"myPlugin">; + static manifest = manifest as PluginManifest<"my-plugin">; async setup() { // Initialize your plugin @@ -69,6 +76,8 @@ class MyPlugin extends Plugin { export const myPlugin = toPlugin(MyPlugin); ``` +JSON is the canonical authoring surface — it is what `appkit plugin sync` reads when aggregating manifests for templates. For the full v2.0 manifest contract (resources, discovery descriptors, post-scaffold steps), see [Plugin manifest](./manifest.md). + ## Config-dependent resources The manifest defines resources as either `required` (always needed) or `optional` (may be needed). diff --git a/docs/docs/plugins/manifest.md b/docs/docs/plugins/manifest.md new file mode 100644 index 000000000..f66d930aa --- /dev/null +++ b/docs/docs/plugins/manifest.md @@ -0,0 +1,255 @@ +--- +sidebar_position: 8 +--- + +# Plugin manifest + +Every plugin ships a `manifest.json` next to its source code. The manifest declares plugin metadata, the Databricks resources the plugin needs, and the post-scaffold instructions surfaced after `databricks apps init`. It is consumed at three stages: + +- **Authoring** — `import manifest from "./manifest.json"` and attach it to the `Plugin` subclass via `static manifest`. +- **Sync** — `appkit plugin sync --write` aggregates manifests from installed packages and local plugins into `appkit.plugins.json`. +- **Init** — `databricks apps init` reads `appkit.plugins.json` to drive plugin selection, resource prompts, and `.env` / `databricks.yml` / `app.yaml` generation. + +This page documents the **v2.0** manifest contract. JSON Schema is published at `https://databricks.github.io/appkit/schemas/plugin-manifest.schema.json`; reference it via `$schema` for editor validation. + +## Recommended pattern + +Author the manifest as JSON, import it into the plugin module, and assert the type: + +```typescript +// packages/my-plugin/src/index.ts +import { Plugin, toPlugin } from "@databricks/appkit"; +import type { PluginManifest } from "@databricks/appkit"; +import manifest from "./manifest.json"; + +class MyPlugin extends Plugin { + static manifest = manifest as PluginManifest<"my-plugin">; + // ... +} + +export const myPlugin = toPlugin(MyPlugin); +``` + +```json +// packages/my-plugin/src/manifest.json +{ + "$schema": "https://databricks.github.io/appkit/schemas/plugin-manifest.schema.json", + "name": "my-plugin", + "displayName": "My Plugin", + "description": "A custom plugin", + "resources": { + "required": [], + "optional": [] + } +} +``` + +JSON is the canonical authoring surface — it is what `appkit plugin sync` reads. JS manifests (`manifest.js` / `manifest.cjs`) are ignored by default and require `--allow-js-manifest` to opt in (executes plugin code; trust required). For end-to-end CLI behavior, see [Plugin management](./plugin-management.md). + +## Required fields + +| Field | Type | Notes | +|-------|------|-------| +| `name` | `string` | Plugin identifier. Lowercase, starts with a letter, `[a-z0-9-]` only. | +| `displayName` | `string` | Shown in UI and CLI prompts. | +| `description` | `string` | Brief summary. | +| `resources.required` | `ResourceRequirement[]` | Resources the plugin cannot run without. | +| `resources.optional` | `ResourceRequirement[]` | Resources that enhance behavior but are not mandatory. | + +## Resources + +A resource requirement declares one Databricks resource the plugin depends on. The shape is keyed by `type`; each type fixes its valid `permission` values (validated by the schema as a discriminated union): + +| `type` | Permissions | +|--------|-------------| +| `secret` | `READ`, `WRITE`, `MANAGE` | +| `job` | `CAN_VIEW`, `CAN_MANAGE_RUN`, `CAN_MANAGE` | +| `sql_warehouse` | `CAN_USE`, `CAN_MANAGE` | +| `serving_endpoint` | `CAN_VIEW`, `CAN_QUERY`, `CAN_MANAGE` | +| `volume` | `READ_VOLUME`, `WRITE_VOLUME` | +| `vector_search_index` | `SELECT` | +| `uc_function` | `EXECUTE` | +| `uc_connection` | `USE_CONNECTION` | +| `database` | `CAN_CONNECT_AND_CREATE` | +| `postgres` | `CAN_CONNECT_AND_CREATE` | +| `genie_space` | `CAN_VIEW`, `CAN_RUN`, `CAN_EDIT`, `CAN_MANAGE` | +| `experiment` | `CAN_READ`, `CAN_EDIT`, `CAN_MANAGE` | +| `app` | `CAN_USE` | + +Every requirement has: + +- `alias` — human-readable label used in UI / CLI output. +- `resourceKey` — stable machine key (`[a-z][a-z0-9-]*`). Used for deduplication, env naming, and references in `app.yaml`. **Identity is keyed on `resourceKey`, not `alias`.** +- `description` — explains *why* this resource is needed; surfaces in interactive prompts. +- `fields` — map of field name → field entry (see below). At least one entry when present. +- `permission` — must match the type's allowed enum. + +Single-value resource types (e.g. `sql_warehouse`) typically declare one field (`id`). Multi-value types (e.g. `secret`, `database`) declare several (`scope` + `key`, `instance_name` + `database_name`). + +### Field entry + +```json +{ + "id": { + "env": "DATABRICKS_WAREHOUSE_ID", + "description": "SQL Warehouse ID", + "examples": ["1234abcd5678efgh"], + "discovery": { "type": "kind", "resourceKind": "warehouse" } + } +} +``` + +| Property | Description | +|----------|-------------| +| `env` | Environment variable name written to `.env` and `app.yaml`. Must match `^[A-Z][A-Z0-9_]*$`. | +| `description` | Shown in interactive prompts and bundle variable descriptions. | +| `examples` | Sample values shown in field descriptions. | +| `localOnly` | When `true`, the field is generated for local `.env` only — the Databricks Apps platform auto-injects it at deploy time, so it is excluded from `app.yaml` and `databricks.yml`. | +| `bundleIgnore` | Excluded from `databricks.yml` variables (still written to `.env`). | +| `value` | Static default value. | +| `resolve` | CLI-side resolver name, formatted `:` (e.g. `postgres:host`). The CLI populates the value from API calls during init. | +| `discovery` | Describes how the CLI lists candidate values — see below. | + +### Configuration-dependent resources + +The manifest distinguishes `required` from `optional` for static analysis. When a resource only becomes required based on the plugin's runtime config, list it under `optional` in the manifest and override at runtime via a static `getResourceRequirements(config)` method on the plugin class. See [Creating custom plugins](./custom-plugins.md#config-dependent-resources). + +## Resource discovery + +Discovery describes how the CLI offers candidate values for a field during interactive init. There are two variants under `discovery`, discriminated by `type`: + +### `kind` variant (preferred) + +```json +{ + "discovery": { + "type": "kind", + "resourceKind": "warehouse" + } +} +``` + +The `kind` variant references a well-known Databricks resource kind for which AppKit owns the listing command and response shape. This is the preferred form for first-party Databricks resources — plugin authors declare *what* to list, and AppKit owns *how* to list it. + +Supported `resourceKind` values: + +| `resourceKind` | Listed via | +|----------------|------------| +| `warehouse` | `databricks warehouses list` | +| `genie_space` | `databricks genie list-spaces` | +| `volume` | `databricks volumes list {catalog} {schema}` | +| `postgres_project` | `databricks postgres list-projects` | +| `postgres_branch` | `databricks postgres list-branches {project}` | +| `postgres_database` | `databricks postgres list-databases {branch}` | + +Supported options on the `kind` variant: + +| Property | Description | +|----------|-------------| +| `select` | Field name in the parsed CLI response used as the selected value (e.g. `"id"`, `"name"`, `"full_name"`). Defaults to the kind's natural identifier. | +| `display` | Field name shown to the user during selection. Defaults to `select`. | +| `dependsOn` | Name of a sibling field within the same resource that must resolve first (see [Field dependencies](#field-dependencies)). | +| `shortcut` | Single-value fast-path command that returns exactly one value, skipping interactive selection. | + +### `cli` variant (escape hatch) + +For resources outside the `kind` map, fall back to the `cli` variant: + +```json +{ + "discovery": { + "type": "cli", + "cliCommand": "databricks custom-resource list --profile --output json", + "selectField": ".id", + "displayField": ".name" + } +} +``` + +| Property | Description | +|----------|-------------| +| `cliCommand` | Full Databricks CLI command. **Must include the literal `` placeholder** — the runner substitutes the user's CLI profile. Shell metacharacters (`;`, `\|`, `&`, `` ` ``, `$`, newlines) are rejected — executors pass arguments via argv, never `shell-exec` the string. | +| `selectField` | jq-style path to the field used as the selected value (e.g. `.id`, `.name`). | +| `displayField` | jq-style path to the field shown to the user. Defaults to `selectField`. | +| `dependsOn` | Sibling field that must resolve first. | +| `shortcut` | Single-value fast-path command. Same metacharacter restriction as `cliCommand`. | + +The `cli` variant is intentionally minimal and may tighten in future versions. **Prefer the `kind` variant** for any resource AppKit knows about; it gives you a single source of truth for command + unwrap rules and guarantees forward-compat as AppKit refines the discovery contract. + +### Field dependencies + +When listing one resource depends on another (e.g. listing volumes requires a catalog and schema; listing Postgres branches requires a project), use `dependsOn` to declare ordering: + +```json +{ + "fields": { + "project": { + "discovery": { "type": "kind", "resourceKind": "postgres_project", "select": "name" } + }, + "branch": { + "discovery": { + "type": "kind", + "resourceKind": "postgres_branch", + "select": "name", + "dependsOn": "project" + } + } + } +} +``` + +`dependsOn` references a sibling field name within the same resource. The CLI prompts in dependency order and substitutes the resolved value into the parent command (e.g. `{project}` in `databricks postgres list-branches {project}`). + +The schema validates the dependency graph at parse time: + +- Dangling references (`dependsOn` pointing at a non-existent sibling) are rejected. +- Cycles are rejected with the chain listed (`a → b → a`). + +For `volume`, AppKit cannot auto-discover the `{catalog}` and `{schema}` parents — there is no UC API the CLI runs by default. The template manifest carries a scaffolding rule that requires the runner to prompt for catalog and schema before listing volumes. + +## Post-scaffold steps + +`postScaffold` is an ordered array of instructions surfaced to the user after `databricks apps init`. Use it to communicate work the CLI cannot automate — things like provisioning a workspace resource, running migrations, or setting up an external integration. + +```json +{ + "postScaffold": [ + { + "instruction": "Run database migrations to initialize your Lakebase schema.", + "required": true + }, + { + "instruction": "Verify local connectivity using: PGHOST= PGDATABASE= psql", + "required": false + } + ] +} +``` + +| Property | Description | +|----------|-------------| +| `instruction` | The user-facing message. Required, must be non-empty. | +| `required` | When `true`, the step must be completed for the plugin to function correctly. Optional; defaults to advisory. | + +Array order is the display order. Steps are propagated unchanged from the plugin manifest into the synced template manifest. + +## Optional fields + +| Field | Description | +|-------|-------------| +| `author` | Author name or organization. | +| `version` | Plugin version, semver format (`X.Y.Z` or `X.Y.Z-prerelease`). | +| `repository` | URL to the plugin source. | +| `keywords` | Discovery keywords. | +| `license` | SPDX identifier. | +| `onSetupMessage` | One-shot message displayed after init. Use for short hints; prefer `postScaffold` for actionable steps. | +| `hidden` | When `true`, the plugin is excluded from the synced template manifest. | +| `stability` | `"beta"` or `"ga"`. Beta plugins may break across minor releases — see [Plugin stability tiers](./stability.md). | +| `config.schema` | JSON Schema for the plugin's runtime config (used by the type generator and for validation). | + +## See also + +- [Creating custom plugins](./custom-plugins.md) — building a plugin from scratch. +- [Plugin management](./plugin-management.md) — `appkit plugin sync`, `create`, `validate`, `add-resource`. +- [Templates](../development/templates.md) — how the synced template manifest drives `databricks apps init`. +- [`PluginManifest` API reference](../api/appkit/Interface.PluginManifest.md) — TypeScript type.