Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/mcp-pilot-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -371,7 +371,7 @@ NitroX is Provar's Hybrid Model for locators — it maps Salesforce component-ba

### Scenario 9 (requires API key): AI Test Generation from User Story

**Goal:** Demonstrate the full Phase 2 AI-assisted test generation loop: org metadata → corpus retrieval → LLM synthesis → generate + validate.
**Goal:** Demonstrate the full AI-assisted test generation loop: org metadata → corpus retrieval → LLM synthesis → generate + validate.

**Setup:**

Expand Down
87 changes: 79 additions & 8 deletions docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ The Provar DX CLI ships with a built-in **Model Context Protocol (MCP) server**
- [provar.qualityhub.defect.create](#provarqualityhubdefectcreate)
- [provar.testrun.report.locate](#provartestrunreportlocate)
- [provar.testrun.rca](#provartestrunrca)
- [provar.testcase.step.edit](#provartestcasestepedit)
- [provar.testplan.add-instance](#provartestplanadinstance)
- [provar.testplan.create-suite](#provartestplancreatetsuite)
- [provar.testplan.remove-instance](#provartestplanremoveinstance)
Expand Down Expand Up @@ -1335,12 +1336,15 @@ Uses a 4-step resolution algorithm (explicit path → `~/.sf/config.json` → `p

Analyse a completed test run and return a structured Root Cause Analysis report. Reads `JUnit.xml`, classifies each failure into a root cause category, extracts page object and operation names, and flags pre-existing failures across prior Increment runs.

| Input | Type | Required | Description |
| -------------- | ------- | -------- | --------------------------------------------------------- |
| `project_path` | string | yes | Absolute path to the Provar project root |
| `results_path` | string | no | Explicit results directory override |
| `run_index` | integer | no | Specific Increment run to analyse (default: latest) |
| `locate_only` | boolean | no | Skip parsing; return artifact paths only (default: false) |
| Input | Type | Required | Description |
| -------------- | ------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `project_path` | string | yes | Absolute path to the Provar project root |
| `results_path` | string | no | Explicit results directory override; must be within `--allowed-paths` if provided |
| `run_index` | integer | no | Specific Increment run to analyse (default: latest) |
| `locate_only` | boolean | no | Skip parsing; return artifact paths only (default: false) |
| `mode` | string | no | `"rca"` (default) — full classification report. `"failures"` — lightweight `[{ testItemId, title, errorMessage }]` array, no RCA fields. |

**mode=rca output fields:**

| Output field | Description |
| ----------------------- | ------------------------------------------------------------------------------- |
Expand All @@ -1352,7 +1356,17 @@ Analyse a completed test run and return a structured Root Cause Analysis report.
| `infrastructure_issues` | Recommendations for infra-category failures (credential, driver, license, etc.) |
| `recommendations` | Deduplicated list of all recommended actions |

**`FailureReport` fields:**
**mode=failures output fields:**

| Output field | Description |
| ----------------- | -------------------------------------------------------------------- |
| `results_dir` | Resolved results directory |
| `failures` | `Array<{ testItemId: string, title: string, errorMessage: string }>` |
| `details.warning` | Set when `JUnit.xml` is absent; `failures` will be empty |

Use `mode="failures"` when you only need the list of failing test case names without loading the full HTML report. Use `mode="rca"` (default) for root-cause classification and fix recommendations.

**`FailureReport` fields (mode=rca only):**

| Field | Description |
| --------------------- | -------------------------------------------------------- |
Expand All @@ -1372,7 +1386,64 @@ Analyse a completed test run and return a structured Root Cause Analysis report.

Salesforce DML error categories (`SALESFORCE_*`) represent test-data failures — they appear in `failures[].root_cause_category` but are **not** included in `infrastructure_issues`.

**Error codes:** `RESULTS_NOT_CONFIGURED`
**Error codes:** `RESULTS_NOT_CONFIGURED`, `PATH_NOT_ALLOWED`, `PATH_TRAVERSAL`

---

### `provar.testcase.step.edit`

Atomically add or remove a single step (`<apiCall>`) in a Provar XML test case file. Writes a `.bak` backup before mutating, runs structural validation after the edit, and automatically restores the backup if validation fails.

Prerequisites: the test case file must exist and be valid XML with a `<testCase><steps>` structure.

| Input | Type | Required | Description |
| --------------------- | ------- | -------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `test_case_path` | string | yes | Absolute path to the `.testcase` file; must be within `--allowed-paths` |
| `mode` | string | yes | `"remove"` — delete a step; `"add"` — insert a new step |
| `test_item_id` | string | yes | For `remove`: `testItemId` of the step to delete. For `add`: `testItemId` of the anchor step. |
| `position` | string | no (add only) | `"before"` or `"after"` relative to the anchor step (default: `"after"`) |
| `step_xml` | string | yes (add only) | The `<apiCall ...>...</apiCall>` XML fragment for the new step. Must be well-formed and contain an `<apiCall>` element. |
| `validate_after_edit` | boolean | no | Run structural validation after the mutation; restores backup on failure (default: `true`) |

| Output field | Description |
| -------------- | ---------------------------------------------------------- |
| `success` | `true` on successful mutation |
| `test_item_id` | The `test_item_id` that was targeted |
| `mode` | `"remove"` or `"add"` |
| `validation` | `TestCaseValidationResult` when `validate_after_edit=true` |

**Error codes:**

| Code | Meaning |
| ------------------------ | ------------------------------------------------------------------------------------------------- |
| `STEP_NOT_FOUND` | No step with the given `testItemId` found; `details.all_test_item_ids` lists every ID in the file |
| `INVALID_STEP_XML` | `step_xml` failed XML parsing or contains no `<apiCall>` element; file is not modified |
| `INVALID_XML_AFTER_EDIT` | Post-mutation validation failed; original file has been restored from backup |
| `FILE_NOT_FOUND` | `test_case_path` does not exist |
| `MISSING_INPUT` | `step_xml` is required for `mode=add` but was not provided |
| `PATH_NOT_ALLOWED` | `test_case_path` or its `.bak` path is outside `--allowed-paths` |

**Example — remove step 3:**

```json
{
"test_case_path": "/projects/myapp/tests/Login.testcase",
"mode": "remove",
"test_item_id": "3"
}
```

**Example — insert a Sleep step after step 2:**

```json
{
"test_case_path": "/projects/myapp/tests/Login.testcase",
"mode": "add",
"test_item_id": "2",
"position": "after",
"step_xml": "<apiCall apiId=\"com.provar.plugins.bundled.apis.control.Sleep\" testItemId=\"99\" guid=\"550e8400-e29b-41d4-a716-446655440099\" name=\"Wait 2s\"><arguments><argument apiId=\"sleepTime\" value=\"2000\"/></arguments></apiCall>"
}
```

---

Expand Down
12 changes: 10 additions & 2 deletions scripts/mcp-smoke.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -360,6 +360,14 @@ async function runTests() {
// TMP has no .testproject → CONNECTION_FILE_NOT_FOUND result (not a protocol error)
await callTool('provar.connection.list', { project_path: TMP });

// ── 50. provar.testcase.step.edit ─────────────────────────────────────────
// TMP/nonexistent.testcase does not exist → FILE_NOT_FOUND result
await callTool('provar.testcase.step.edit', {
test_case_path: path.join(TMP, 'nonexistent.testcase'),
mode: 'remove',
test_item_id: '1',
});

server.stdin.end();
}

Expand All @@ -368,8 +376,8 @@ async function runTests() {
// ----------------------------------------------------------------------------
server.on('close', () => {
clearTimeout(overallTimer);
// initialize + tools/list + 38 tools + prompts/list + 8 prompts/get (setup excluded from default count)
const TOTAL_EXPECTED = 49 + (INCLUDE_SETUP ? 1 : 0);
// initialize + tools/list + 39 tools + prompts/list + 8 prompts/get (setup excluded from default count)
const TOTAL_EXPECTED = 50 + (INCLUDE_SETUP ? 1 : 0);
let passed = 0;
let failed = 0;

Expand Down
2 changes: 1 addition & 1 deletion src/commands/provar/auth/login.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ import {
Messages.importMessagesDirectoryFromMetaUrl(import.meta.url);
const messages = Messages.loadMessages('@provartesting/provardx-cli', 'sf.provar.auth.login');

// Production values bundled from Phase 2 handoff (2026-04-11).
// Production values bundled at auth handoff (2026-04-11).
// Override via PROVAR_COGNITO_DOMAIN / PROVAR_COGNITO_CLIENT_ID for non-prod environments.
const DEFAULT_COGNITO_DOMAIN = 'us-east-1xpfwzwmop.auth.us-east-1.amazoncognito.com';
const DEFAULT_CLIENT_ID = '29cs1a784r4cervmth8ugbkkri';
Expand Down
Loading
Loading