diff --git a/docs/mcp-pilot-guide.md b/docs/mcp-pilot-guide.md index 2ca3fc7..bc7283e 100644 --- a/docs/mcp-pilot-guide.md +++ b/docs/mcp-pilot-guide.md @@ -371,7 +371,7 @@ NitroX is Provar's Hybrid Model for locators — it maps Salesforce component-ba ### Scenario 9 (requires API key): AI Test Generation from User Story -**Goal:** Demonstrate the full Phase 2 AI-assisted test generation loop: org metadata → corpus retrieval → LLM synthesis → generate + validate. +**Goal:** Demonstrate the full AI-assisted test generation loop: org metadata → corpus retrieval → LLM synthesis → generate + validate. **Setup:** diff --git a/docs/mcp.md b/docs/mcp.md index 695390c..fa3dce8 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -46,6 +46,7 @@ The Provar DX CLI ships with a built-in **Model Context Protocol (MCP) server** - [provar.qualityhub.defect.create](#provarqualityhubdefectcreate) - [provar.testrun.report.locate](#provartestrunreportlocate) - [provar.testrun.rca](#provartestrunrca) + - [provar.testcase.step.edit](#provartestcasestepedit) - [provar.testplan.add-instance](#provartestplanadinstance) - [provar.testplan.create-suite](#provartestplancreatetsuite) - [provar.testplan.remove-instance](#provartestplanremoveinstance) @@ -1335,12 +1336,15 @@ Uses a 4-step resolution algorithm (explicit path → `~/.sf/config.json` → `p Analyse a completed test run and return a structured Root Cause Analysis report. Reads `JUnit.xml`, classifies each failure into a root cause category, extracts page object and operation names, and flags pre-existing failures across prior Increment runs. -| Input | Type | Required | Description | -| -------------- | ------- | -------- | --------------------------------------------------------- | -| `project_path` | string | yes | Absolute path to the Provar project root | -| `results_path` | string | no | Explicit results directory override | -| `run_index` | integer | no | Specific Increment run to analyse (default: latest) | -| `locate_only` | boolean | no | Skip parsing; return artifact paths only (default: false) | +| Input | Type | Required | Description | +| -------------- | ------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------- | +| `project_path` | string | yes | Absolute path to the Provar project root | +| `results_path` | string | no | Explicit results directory override; must be within `--allowed-paths` if provided | +| `run_index` | integer | no | Specific Increment run to analyse (default: latest) | +| `locate_only` | boolean | no | Skip parsing; return artifact paths only (default: false) | +| `mode` | string | no | `"rca"` (default) — full classification report. `"failures"` — lightweight `[{ testItemId, title, errorMessage }]` array, no RCA fields. | + +**mode=rca output fields:** | Output field | Description | | ----------------------- | ------------------------------------------------------------------------------- | @@ -1352,7 +1356,17 @@ Analyse a completed test run and return a structured Root Cause Analysis report. | `infrastructure_issues` | Recommendations for infra-category failures (credential, driver, license, etc.) | | `recommendations` | Deduplicated list of all recommended actions | -**`FailureReport` fields:** +**mode=failures output fields:** + +| Output field | Description | +| ----------------- | -------------------------------------------------------------------- | +| `results_dir` | Resolved results directory | +| `failures` | `Array<{ testItemId: string, title: string, errorMessage: string }>` | +| `details.warning` | Set when `JUnit.xml` is absent; `failures` will be empty | + +Use `mode="failures"` when you only need the list of failing test case names without loading the full HTML report. Use `mode="rca"` (default) for root-cause classification and fix recommendations. + +**`FailureReport` fields (mode=rca only):** | Field | Description | | --------------------- | -------------------------------------------------------- | @@ -1372,7 +1386,64 @@ Analyse a completed test run and return a structured Root Cause Analysis report. Salesforce DML error categories (`SALESFORCE_*`) represent test-data failures — they appear in `failures[].root_cause_category` but are **not** included in `infrastructure_issues`. -**Error codes:** `RESULTS_NOT_CONFIGURED` +**Error codes:** `RESULTS_NOT_CONFIGURED`, `PATH_NOT_ALLOWED`, `PATH_TRAVERSAL` + +--- + +### `provar.testcase.step.edit` + +Atomically add or remove a single step (``) in a Provar XML test case file. Writes a `.bak` backup before mutating, runs structural validation after the edit, and automatically restores the backup if validation fails. + +Prerequisites: the test case file must exist and be valid XML with a `` structure. + +| Input | Type | Required | Description | +| --------------------- | ------- | -------------- | ----------------------------------------------------------------------------------------------------------------------- | +| `test_case_path` | string | yes | Absolute path to the `.testcase` file; must be within `--allowed-paths` | +| `mode` | string | yes | `"remove"` — delete a step; `"add"` — insert a new step | +| `test_item_id` | string | yes | For `remove`: `testItemId` of the step to delete. For `add`: `testItemId` of the anchor step. | +| `position` | string | no (add only) | `"before"` or `"after"` relative to the anchor step (default: `"after"`) | +| `step_xml` | string | yes (add only) | The `...` XML fragment for the new step. Must be well-formed and contain an `` element. | +| `validate_after_edit` | boolean | no | Run structural validation after the mutation; restores backup on failure (default: `true`) | + +| Output field | Description | +| -------------- | ---------------------------------------------------------- | +| `success` | `true` on successful mutation | +| `test_item_id` | The `test_item_id` that was targeted | +| `mode` | `"remove"` or `"add"` | +| `validation` | `TestCaseValidationResult` when `validate_after_edit=true` | + +**Error codes:** + +| Code | Meaning | +| ------------------------ | ------------------------------------------------------------------------------------------------- | +| `STEP_NOT_FOUND` | No step with the given `testItemId` found; `details.all_test_item_ids` lists every ID in the file | +| `INVALID_STEP_XML` | `step_xml` failed XML parsing or contains no `` element; file is not modified | +| `INVALID_XML_AFTER_EDIT` | Post-mutation validation failed; original file has been restored from backup | +| `FILE_NOT_FOUND` | `test_case_path` does not exist | +| `MISSING_INPUT` | `step_xml` is required for `mode=add` but was not provided | +| `PATH_NOT_ALLOWED` | `test_case_path` or its `.bak` path is outside `--allowed-paths` | + +**Example — remove step 3:** + +```json +{ + "test_case_path": "/projects/myapp/tests/Login.testcase", + "mode": "remove", + "test_item_id": "3" +} +``` + +**Example — insert a Sleep step after step 2:** + +```json +{ + "test_case_path": "/projects/myapp/tests/Login.testcase", + "mode": "add", + "test_item_id": "2", + "position": "after", + "step_xml": "" +} +``` --- diff --git a/scripts/mcp-smoke.cjs b/scripts/mcp-smoke.cjs index 42ee048..06b709f 100644 --- a/scripts/mcp-smoke.cjs +++ b/scripts/mcp-smoke.cjs @@ -360,6 +360,14 @@ async function runTests() { // TMP has no .testproject → CONNECTION_FILE_NOT_FOUND result (not a protocol error) await callTool('provar.connection.list', { project_path: TMP }); + // ── 50. provar.testcase.step.edit ───────────────────────────────────────── + // TMP/nonexistent.testcase does not exist → FILE_NOT_FOUND result + await callTool('provar.testcase.step.edit', { + test_case_path: path.join(TMP, 'nonexistent.testcase'), + mode: 'remove', + test_item_id: '1', + }); + server.stdin.end(); } @@ -368,8 +376,8 @@ async function runTests() { // ---------------------------------------------------------------------------- server.on('close', () => { clearTimeout(overallTimer); - // initialize + tools/list + 38 tools + prompts/list + 8 prompts/get (setup excluded from default count) - const TOTAL_EXPECTED = 49 + (INCLUDE_SETUP ? 1 : 0); + // initialize + tools/list + 39 tools + prompts/list + 8 prompts/get (setup excluded from default count) + const TOTAL_EXPECTED = 50 + (INCLUDE_SETUP ? 1 : 0); let passed = 0; let failed = 0; diff --git a/src/commands/provar/auth/login.ts b/src/commands/provar/auth/login.ts index fb0ae1e..6d4bc7b 100644 --- a/src/commands/provar/auth/login.ts +++ b/src/commands/provar/auth/login.ts @@ -20,7 +20,7 @@ import { Messages.importMessagesDirectoryFromMetaUrl(import.meta.url); const messages = Messages.loadMessages('@provartesting/provardx-cli', 'sf.provar.auth.login'); -// Production values bundled from Phase 2 handoff (2026-04-11). +// Production values bundled at auth handoff (2026-04-11). // Override via PROVAR_COGNITO_DOMAIN / PROVAR_COGNITO_CLIENT_ID for non-prod environments. const DEFAULT_COGNITO_DOMAIN = 'us-east-1xpfwzwmop.auth.us-east-1.amazoncognito.com'; const DEFAULT_CLIENT_ID = '29cs1a784r4cervmth8ugbkkri'; diff --git a/src/mcp/rules/provar_best_practices_rules.json b/src/mcp/rules/provar_best_practices_rules.json index 298b39c..77f98d2 100644 --- a/src/mcp/rules/provar_best_practices_rules.json +++ b/src/mcp/rules/provar_best_practices_rules.json @@ -34,9 +34,7 @@ "category": "XMLSchema", "name": "Test case root element must be testCase", "description": "The root XML element must be . Any other root element prevents the test from being imported into Provar.", - "appliesTo": [ - "TestCase" - ], + "appliesTo": ["TestCase"], "severity": "critical", "weight": 10, "recommendation": "Ensure the root element is with proper namespace and attributes.", @@ -52,9 +50,7 @@ "category": "XMLSchema", "name": "Test case must have valid identifier", "description": "Test case must have one of: 'id' (recommended), 'guid' (Provar V3), or 'registryId' (legacy) attribute. Missing identifier prevents test case import.", - "appliesTo": [ - "TestCase" - ], + "appliesTo": ["TestCase"], "severity": "critical", "weight": 10, "recommendation": "Add 'id' attribute to element (e.g., id=\"1\"), or use 'guid' for Provar V3 compatibility.", @@ -70,9 +66,7 @@ "category": "XMLSchema", "name": "Test case should have failureBehaviour attribute", "description": "The 'failureBehaviour' attribute is recommended for new tests (default: 'Continue'). While not required, it makes test behavior explicit.", - "appliesTo": [ - "TestCase" - ], + "appliesTo": ["TestCase"], "severity": "info", "weight": 1, "recommendation": "Add failureBehaviour=\"Continue\" to element for explicit failure handling.", @@ -88,9 +82,7 @@ "category": "XMLSchema", "name": "Consider migrating from registryId to id or guid", "description": "Using legacy 'registryId' attribute. Consider migrating to 'id' (simple) or 'guid' (Provar V3) for better compatibility.", - "appliesTo": [ - "TestCase" - ], + "appliesTo": ["TestCase"], "severity": "info", "weight": 1, "recommendation": "Replace registryId with 'id' or 'guid' attribute for Provar V3 compatibility.", @@ -106,9 +98,7 @@ "category": "XMLSchema", "name": "Value elements must not use text attribute", "description": "Content must be inside element, not as text=\"...\" attribute. Using text attribute causes XML parsing errors.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 10, "recommendation": "Change to content", @@ -124,9 +114,7 @@ "category": "XMLSchema", "name": "URI attributes must properly encode ampersands", "description": "Ampersands in URI attributes must be encoded as & for valid XML. Unencoded & causes XML parsing errors.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 10, "recommendation": "Replace & with & in all uri=\"...\" attributes (e.g., uri=\"sf:ui:binding:object?object=Lead&action=View\")", @@ -142,9 +130,7 @@ "category": "XMLSchema", "name": "Test case must have steps element", "description": "Test case must contain a element with at least one apiCall. Missing steps element prevents test case from loading.", - "appliesTo": [ - "TestCase" - ], + "appliesTo": ["TestCase"], "severity": "critical", "weight": 10, "recommendation": "Add element containing apiCall steps to the test case.", @@ -160,9 +146,7 @@ "category": "XMLSchema", "name": "Test case should not be empty", "description": "Test case element should contain at least one . Empty test cases have no executable logic.", - "appliesTo": [ - "TestCase" - ], + "appliesTo": ["TestCase"], "severity": "minor", "weight": 2, "recommendation": "Add apiCall steps to implement test logic.", @@ -178,9 +162,7 @@ "category": "XMLSchema", "name": "API identifier must be a valid Provar API", "description": "The apiId attribute must reference a real Provar API. Unknown or hallucinated apiIds (like 'ApexDescribeObject') cause complete test step failures because Provar cannot find the API implementation.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 10, "recommendation": "Use only valid Provar apiIds. Common corrections: ApexDescribeObject→ApexReadObject, ApexQueryObject→ApexSoqlQuery, ApexInsertObject→ApexCreateObject. Refer to provar_test_step_schema.json for the complete list of valid APIs.", @@ -195,10 +177,7 @@ "category": "NamingConventions", "name": "Folder names are modular and title-cased", "description": "Test folders must be concise, modular, alphabetic, and use Title Case with spaces between words.", - "appliesTo": [ - "Folder", - "TestCase" - ], + "appliesTo": ["Folder", "TestCase"], "severity": "major", "weight": 5, "recommendation": "Rename folder to be concise, alphabetic only, and in Title Case (e.g. 'Test Case Design').", @@ -214,9 +193,7 @@ "category": "NamingConventions", "name": "Test case names use consistent naming convention", "description": "Test case names must follow a consistent naming pattern. NOTE: This rule only applies to test cases with an explicit 'name' attribute (typically callable test cases). The 'id' attribute can be any format.", - "appliesTo": [ - "TestCase" - ], + "appliesTo": ["TestCase"], "severity": "minor", "weight": 2, "recommendation": "Use a consistent naming style for callable test case names. Examples: 'CreateAndConvertLead', 'createAndConvertLead', 'Create And Convert Lead'.", @@ -233,9 +210,7 @@ "category": "NamingConventions", "name": "Parameters and variables use camelCase", "description": "Test parameters, variables, and return values must be camelCase and unique within their scope.", - "appliesTo": [ - "Parameter" - ], + "appliesTo": ["Parameter"], "severity": "major", "weight": 5, "recommendation": "Rename parameters and variables to camelCase and ensure no duplicates in the same scope.", @@ -251,10 +226,7 @@ "category": "NamingConventions", "name": "Page Objects use PascalCase", "description": "Page Object names must follow PascalCase and represent a single logical page or screen.", - "appliesTo": [ - "PageObject", - "TestCase" - ], + "appliesTo": ["PageObject", "TestCase"], "severity": "major", "weight": 5, "recommendation": "Rename Page Object to PascalCase (e.g. 'AccountDetailsPage').", @@ -270,10 +242,7 @@ "category": "NamingConventions", "name": "Field names use camelCase", "description": "Field mapping names must be camelCase for consistency and readability.", - "appliesTo": [ - "Field", - "TestCase" - ], + "appliesTo": ["Field", "TestCase"], "severity": "minor", "weight": 2, "recommendation": "Rename field mapping to camelCase (e.g. 'billingStreet').", @@ -289,9 +258,7 @@ "category": "StructureAndGrouping", "name": "All steps are inside Group steps or BDD structure", "description": "Every test step must be contained within a Group Step, BDD design step (Given/When/Then/And/But), or Finally block to improve readability and maintenance.", - "appliesTo": [ - "TestCase" - ], + "appliesTo": ["TestCase"], "severity": "minor", "weight": 2, "recommendation": "Wrap root-level steps into logically named Group Steps, BDD steps (Given/When/Then), or Finally blocks.", @@ -308,9 +275,7 @@ "category": "StructureAndGrouping", "name": "Test case has top-level summary", "description": "Each test case must have a summary describing its purpose and usage.", - "appliesTo": [ - "TestCase" - ], + "appliesTo": ["TestCase"], "severity": "info", "weight": 1, "recommendation": "Add a concise summary describing scenario, preconditions, and outcome.", @@ -325,19 +290,14 @@ "category": "StructureAndGrouping", "name": "Custom step names for UI asserts and sets", "description": "UI Assert, Set Values, and long SOQL query steps must be renamed to be descriptive.", - "appliesTo": [ - "Step", - "TestCase" - ], + "appliesTo": ["Step", "TestCase"], "severity": "minor", "weight": 2, "recommendation": "Rename the step to describe what is being asserted or set (e.g. 'assertAccountStatusIsActive').", "check": { "type": "semanticPattern", "target": "step", - "appliesWhen": [ - "step.type in ['UIAssert','SetValues','SOQL']" - ], + "appliesWhen": ["step.type in ['UIAssert','SetValues','SOQL']"], "mustHaveCustomName": true }, "source": "README: Naming Conventions - Test Step Names" @@ -347,9 +307,7 @@ "category": "DataDrivenTesting", "name": "Excel headers match field label or API name", "description": "Excel/data source column headers must match either the field label or API name.", - "appliesTo": [ - "DataSource" - ], + "appliesTo": ["DataSource"], "severity": "major", "weight": 5, "recommendation": "Rename Excel column headers to match Salesforce field labels or API names.", @@ -365,10 +323,7 @@ "category": "DataDrivenTesting", "name": "No Excel functions in data", "description": "Excel test data must be static; functions are not allowed in cells.", - "appliesTo": [ - "DataSource", - "TestCase" - ], + "appliesTo": ["DataSource", "TestCase"], "severity": "major", "weight": 5, "recommendation": "Replace Excel formulas with static values to keep tests deterministic.", @@ -384,9 +339,7 @@ "category": "DataDrivenTesting", "name": "Variable names must use valid identifiers", "description": "Variable names, field references, and result names must contain only letters, digits, and underscores. Spaces, hyphens, and special characters cause RUNTIME FAILURES in Provar. Examples: {Account.Name} ✅ | {Account.Account Name} ❌ (space will fail)", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Replace spaces and special characters with underscores. Use API field names (e.g., Account.Name) instead of field labels (e.g., Account.Account Name).", @@ -403,10 +356,7 @@ "category": "DataDrivenTesting", "name": "No hardcoded values in steps", "description": "Fields and values used more than once in a test must be parameterized or stored in variables.", - "appliesTo": [ - "Step", - "TestCase" - ], + "appliesTo": ["Step", "TestCase"], "severity": "minor", "weight": 3, "recommendation": "Replace repeated literal values with variables or parameters defined in the test case.", @@ -423,9 +373,7 @@ "category": "ReusabilityAndCallables", "name": "Callable tests reside in Callables folder", "description": "Callable tests must be stored under a designated Callables folder for discoverability.", - "appliesTo": [ - "TestCase" - ], + "appliesTo": ["TestCase"], "severity": "minor", "weight": 2, "recommendation": "Move callable test into 'tests/Callables' (or equivalent) folder.", @@ -440,9 +388,7 @@ "category": "ReusabilityAndCallables", "name": "Callable tests are parameterized", "description": "Callable tests must define input parameters instead of using internal hardcoded values.", - "appliesTo": [ - "TestCase" - ], + "appliesTo": ["TestCase"], "severity": "minor", "weight": 2, "recommendation": "Add input parameters to the callable test and reference them in steps instead of literals.", @@ -458,9 +404,7 @@ "category": "ReusabilityAndCallables", "name": "Callable tests executable in isolation", "description": "Callable tests must be self-contained and executable independently for debugging.", - "appliesTo": [ - "TestCase" - ], + "appliesTo": ["TestCase"], "severity": "minor", "weight": 2, "recommendation": "Ensure callable tests create their own preconditions (records, connections) or accept them as parameters.", @@ -476,9 +420,7 @@ "category": "ConnectionsAndEnvironments", "name": "Admin connection supports Login-As", "description": "There must be an Admin connection configured to perform Login-As for other profiles.", - "appliesTo": [ - "Project" - ], + "appliesTo": ["Project"], "severity": "major", "weight": 5, "recommendation": "Create an Admin connection with 'login as' permissions for target profiles.", @@ -494,9 +436,7 @@ "category": "ConnectionsAndEnvironments", "name": "Connection names should not contain environment specifiers", "description": "Connection names should be environment-agnostic. Avoid embedding UAT, QA, Sandbox, Prod, Production, or Scratch in connection names. Use environment overrides instead.", - "appliesTo": [ - "TestCase" - ], + "appliesTo": ["TestCase"], "severity": "minor", "weight": 2, "recommendation": "Remove environment specifiers from connection names. Create environment-specific connections via Provar environment overrides.", @@ -511,9 +451,7 @@ "category": "TestCaseDesign", "name": "Prefer API for setup where possible", "description": "Test setup steps that only prepare data should use API/SOQL instead of UI flows when feasible.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "minor", "weight": 3, "recommendation": "Replace UI-based record creation for setup with API or SOQL-based data creation where appropriate.", @@ -530,9 +468,7 @@ "category": "TestCaseDesign", "name": "Use Group Steps or BDD structure for logical phases", "description": "Long tests should be organized using Group Steps, BDD design steps (Given/When/Then/And/But), or Finally blocks that represent logical phases of the flow.", - "appliesTo": [ - "TestCase" - ], + "appliesTo": ["TestCase"], "severity": "minor", "weight": 2, "recommendation": "Organize test with Group Steps (e.g. 'Setup', 'Execute Scenario', 'Validate Results'), BDD steps (Given/When/Then), or Finally blocks around related actions.", @@ -550,9 +486,7 @@ "category": "TestCaseDesign", "name": "Disabled test steps should be removed", "description": "Test steps marked with disabled should be removed from the test case to maintain clean, maintainable code. Disabled steps create technical debt and confusion.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "minor", "weight": 2, "recommendation": "Remove disabled test steps entirely. If the step is needed for future use, consider creating a separate test case or adding it when actually required.", @@ -567,9 +501,7 @@ "category": "MaintenanceAndFolders", "name": "Folder-level setup test per application segment", "description": "Each folder representing an application segment must include a setup test case for connections.", - "appliesTo": [ - "Folder" - ], + "appliesTo": ["Folder"], "severity": "minor", "weight": 3, "recommendation": "Add a setup test case that establishes folder-wide connections (even if folder scope bug exists).", @@ -585,9 +517,7 @@ "category": "MaintenanceAndFolders", "name": "Consistent Provar/OS/browser versions", "description": "Execution environments and authors should align on Provar, OS, and browser versions.", - "appliesTo": [ - "Project" - ], + "appliesTo": ["Project"], "severity": "info", "weight": 1, "recommendation": "Document and standardize supported Provar, OS, and browser versions in project configuration.", @@ -602,9 +532,7 @@ "category": "BuildAndCI", "name": "Regression Test Plan exists for CI", "description": "A Regression Test Plan must exist and be referenced by CI builds.", - "appliesTo": [ - "Project" - ], + "appliesTo": ["Project"], "severity": "major", "weight": 5, "recommendation": "Ensure there is a 'Regression' plan and that the CI workflow references it.", @@ -620,9 +548,7 @@ "category": "TestCaseDesign", "name": "Test case has valid identifier", "description": "Test case must have a valid identifier: 'guid' (V3), 'id', or 'registryId' (legacy) attribute.", - "appliesTo": [ - "TestCase" - ], + "appliesTo": ["TestCase"], "severity": "critical", "weight": 8, "recommendation": "Add a valid identifier to the element: guid (preferred), id, or registryId attribute.", @@ -637,9 +563,7 @@ "category": "TestCaseDesign", "name": "Test case has steps element", "description": "Test case must contain a element with at least one step.", - "appliesTo": [ - "TestCase" - ], + "appliesTo": ["TestCase"], "severity": "critical", "weight": 8, "recommendation": "Add a element containing apiCall steps.", @@ -654,9 +578,7 @@ "category": "ConnectionsAndEnvironments", "name": "UiConnect has invalid arguments (ApexConnect arguments used)", "description": "UiConnect steps have different arguments than ApexConnect. UiConnect does NOT support: autoCleanup, enableObjectIdLogging, quickUiLogin, closeAllPrimaryTabs, alreadyOpenBehaviour, lightningMode, uiApplicationName, cleanupConnectionName. Valid UiConnect arguments are: connectionName, connectionId, resultName, resultScope, reuseConnectionName, privateBrowsingMode, webBrowser.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 10, "recommendation": "Remove invalid arguments from UiConnect. For Salesforce connections requiring autoCleanup, use ApexConnect instead of UiConnect. UiConnect is for non-Salesforce UI connections only.", @@ -671,9 +593,7 @@ "category": "ConnectionsAndEnvironments", "name": "Prefer autoCleanup over manual ApexDeleteObject steps", "description": "ApexConnect should use autoCleanup=true instead of manual ApexDeleteObject steps. autoCleanup is preferred because it requires fewer steps, is more reliable (cleanup happens even if test fails), and automatically handles deletion order. Only use manual deletes when records are created indirectly and IDs aren't captured. Note: With multiple ApexConnect steps, autoCleanup only deletes records created using that specific connection's resultName.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "minor", "weight": 2, "recommendation": "Set autoCleanup=true in ApexConnect and remove manual ApexDeleteObject steps. autoCleanup automatically deletes records created via ApexCreateObject and UiWithScreen action=New (when sfUiTargetResultName captures the ID). With multiple connections, each autoCleanup only handles its own connection's records.", @@ -688,9 +608,7 @@ "category": "TestCaseDesign", "name": "First UiWithScreen must use navigate=Always or IfNeccessary", "description": "The first UiWithScreen step must use navigate='Always' or 'IfNeccessary' to ensure proper navigation. Using 'Dont' will cause failures in CLI/debug mode. This rule is skipped for callable tests (visibility='Internal').", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Set navigate='Always' (preferred) or 'IfNeccessary' for the first UiWithScreen step. Never use 'Dont' on the first UiWithScreen unless in a callable test.", @@ -705,9 +623,7 @@ "category": "TestCaseDesign", "name": "First UiWithScreen should prefer navigate=Always over IfNeccessary", "description": "The first UiWithScreen step should use navigate='Always' rather than 'IfNeccessary' for more reliable navigation. While 'IfNeccessary' works, 'Always' provides more consistent behavior. This rule is skipped for callable tests (visibility='Internal').", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "minor", "weight": 2, "recommendation": "Change navigate='IfNeccessary' to navigate='Always' for the first UiWithScreen step for more reliable navigation.", @@ -722,9 +638,7 @@ "category": "TestCaseDesign", "name": "UiWithScreen with navigate=Always for Edit/View must have sfUiTargetObjectId", "description": "When UiWithScreen navigates to an Edit or View screen with navigate='Always', the sfUiTargetObjectId argument must be populated to specify which record to edit or view. Without this, Provar cannot determine which record to navigate to. This does not apply to New screens (which create new records) or List/Home screens.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Add sfUiTargetObjectId argument referencing a previously captured record ID variable (e.g., AccountId from a prior UiWithScreen with action=New or ApexCreateObject).", @@ -739,9 +653,7 @@ "category": "NamingConventions", "name": "ApexConnect resultName is unique", "description": "Each ApexConnect step must have a unique resultName for proper connection tracking.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "minor", "weight": 2, "recommendation": "Use unique resultName values like 'ApexConnection', 'ApexConnection2', etc.", @@ -757,9 +669,7 @@ "category": "TestCaseDesign", "name": "testItemId values are whole numbers", "description": "All testItemId attributes must be whole numbers, not decimals.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Use whole numbers for testItemId (1, 2, 3...) not decimals (1.0, 2.5).", @@ -774,9 +684,7 @@ "category": "DataDrivenTesting", "name": "Variable references use correct syntax", "description": "Variable references must use {VarName[1].Field} syntax, not {VarName.1.Field}.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "minor", "weight": 2, "recommendation": "Use {ResultName[index].FieldName} where index starts at 1, not 0.", @@ -792,9 +700,7 @@ "category": "NamingConventions", "name": "Custom fields end with __c", "description": "Custom field references must end with '__c' suffix.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Add '__c' suffix to custom field names (e.g. 'CustomField__c').", @@ -809,9 +715,7 @@ "category": "TestCaseDesign", "name": "SOQL queries include Id and Name", "description": "SOQL queries should retrieve Id and Name fields by default.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "minor", "weight": 2, "recommendation": "Add 'Id, Name' to SELECT clause for object queries.", @@ -827,9 +731,7 @@ "category": "TestCaseDesign", "name": "ForEach loops have valid source collection", "description": "ForEach steps must specify a valid source collection to iterate over.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 4, "recommendation": "Add 'list' argument to ForEach step referencing a valid collection variable.", @@ -845,9 +747,7 @@ "category": "TestCaseDesign", "name": "If statements have conditions", "description": "If steps must have a boolean condition expression.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add 'condition' argument to If step with valid boolean expression.", @@ -863,9 +763,7 @@ "category": "TestCaseDesign", "name": "While loops have exit conditions", "description": "While loops must have a condition to prevent infinite loops.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add 'condition' argument to While step that will eventually evaluate to false.", @@ -881,9 +779,7 @@ "category": "TestCaseDesign", "name": "ForEach loops have valueName to store current item", "description": "ForEach steps must specify a valueName argument to store the current iteration value.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add 'valueName' argument to ForEach step to name the variable that will hold each item during iteration.", @@ -899,9 +795,7 @@ "category": "TestCaseDesign", "name": "Switch statements have value expression", "description": "Switch steps must specify a value argument to evaluate against cases.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add 'value' argument to Switch step with the expression to evaluate.", @@ -917,9 +811,7 @@ "category": "TestCaseDesign", "name": "Sleep steps have duration specified", "description": "Sleep steps must specify sleepSecs argument with the number of seconds to wait.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add 'sleepSecs' argument to Sleep step with a positive decimal value.", @@ -935,9 +827,7 @@ "category": "TestCaseDesign", "name": "WaitFor steps have condition", "description": "WaitFor steps must specify a condition argument to evaluate each iteration.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add 'condition' argument to WaitFor step with a boolean expression to wait for.", @@ -953,9 +843,7 @@ "category": "TestCaseDesign", "name": "WaitFor steps have max iterations limit", "description": "WaitFor steps must specify maxIterations to prevent infinite waiting.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Add 'maxIterations' argument to WaitFor step to limit wait cycles.", @@ -971,9 +859,7 @@ "category": "TestCaseDesign", "name": "AssertValues has comparisonType", "description": "AssertValues steps must specify a comparisonType argument (EqualTo, Contains, etc.).", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add 'comparisonType' argument with valid value: EqualTo, NotEqualTo, Contains, NotContains, StartsWith, EndsWith, GreaterThan, LessThan, Matches, IsNull, NotNull.", @@ -989,9 +875,7 @@ "category": "TestCaseDesign", "name": "AssertValues has expectedValue", "description": "AssertValues steps must specify an expectedValue argument containing the value to test.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add 'expectedValue' argument with the variable or value being verified.", @@ -1007,9 +891,7 @@ "category": "TestCaseDesign", "name": "AssertValues has actualValue", "description": "AssertValues steps must specify an actualValue argument containing the expected result. NOTE: actualValue CAN be empty when using NotEqualTo to check if a string is NOT blank (common pattern for null/blank checks).", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add 'actualValue' argument with the expected result to compare against. For blank string checks, actualValue can be empty with NotEqualTo comparison.", @@ -1025,9 +907,7 @@ "category": "TestCaseDesign", "name": "Given steps have description", "description": "BDD Given steps must specify a description argument explaining the precondition.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Add 'description' argument to Given step with a clear precondition statement.", @@ -1043,9 +923,7 @@ "category": "TestCaseDesign", "name": "When steps have description", "description": "BDD When steps must specify a description argument explaining the action.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Add 'description' argument to When step with a clear action statement.", @@ -1061,9 +939,7 @@ "category": "TestCaseDesign", "name": "Then steps have description", "description": "BDD Then steps must specify a description argument explaining the expected outcome.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Add 'description' argument to Then step with a clear expected outcome statement.", @@ -1079,9 +955,7 @@ "category": "TestCaseDesign", "name": "ApexSoqlQuery has soqlQuery argument", "description": "ApexSoqlQuery steps must specify a soqlQuery argument with the SOQL statement.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add 'soqlQuery' argument with valid SOQL statement (e.g., SELECT Id, Name FROM Account).", @@ -1097,9 +971,7 @@ "category": "ConnectionsAndEnvironments", "name": "DbConnect has connectionName", "description": "DbConnect steps must specify a connectionName argument identifying the database connection.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add 'connectionName' argument with the database connection identifier from environment settings.", @@ -1115,9 +987,7 @@ "category": "ConnectionsAndEnvironments", "name": "DbConnect has resultName", "description": "DbConnect steps must specify a resultName argument to store the connection reference.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add 'resultName' argument with the variable name to store the connection reference.", @@ -1133,9 +1003,7 @@ "category": "TestCaseDesign", "name": "SqlQuery has query argument", "description": "SqlQuery steps must specify a query argument with the SQL statement.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add 'query' argument with valid SQL statement.", @@ -1151,9 +1019,7 @@ "category": "TestCaseDesign", "name": "SqlQuery has dbConnectionName", "description": "SqlQuery steps must specify a dbConnectionName argument referencing the database connection.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add 'dbConnectionName' argument with the resultName from a DbConnect step.", @@ -1169,9 +1035,7 @@ "category": "ConnectionsAndEnvironments", "name": "WebConnect has connectionName", "description": "WebConnect steps must specify a connectionName argument identifying the web service connection.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add 'connectionName' argument with the web service connection identifier.", @@ -1187,9 +1051,7 @@ "category": "ConnectionsAndEnvironments", "name": "WebConnect has resultName", "description": "WebConnect steps must specify a resultName argument to store the connection reference.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add 'resultName' argument with the variable name to store the connection reference.", @@ -1205,9 +1067,7 @@ "category": "TestCaseDesign", "name": "RestRequest has connectionName", "description": "RestRequest steps must specify a connectionName argument referencing the web service connection.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add 'connectionName' argument with the resultName from a WebConnect step.", @@ -1223,10 +1083,7 @@ "category": "TestCaseDesign", "name": "Variables are defined before use", "description": "Variables referenced in test should be defined via SetValues or result from previous steps.", - "appliesTo": [ - "Step", - "TestCase" - ], + "appliesTo": ["Step", "TestCase"], "severity": "major", "weight": 5, "recommendation": "Define variable with SetValues step or ensure it's populated by previous step (SOQL, CreateObject, etc.).", @@ -1241,9 +1098,7 @@ "category": "ConnectionsAndEnvironments", "name": "Apex API calls reference valid connections", "description": "Apex API operations must reference an existing ApexConnect connection.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Ensure ApexConnect step exists before Apex API calls and connection names match.", @@ -1258,9 +1113,7 @@ "category": "ConnectionsAndEnvironments", "name": "Database operations reference valid connections", "description": "Database API operations must reference an existing DbConnect connection.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Ensure DbConnect step exists before database operations and connection names match.", @@ -1275,9 +1128,7 @@ "category": "ConnectionsAndEnvironments", "name": "UI operations reference valid connections", "description": "UI API operations must reference an existing UiConnect connection.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Ensure UiConnect step exists before UI operations and connection names match.", @@ -1292,9 +1143,7 @@ "category": "ConnectionsAndEnvironments", "name": "uiConnectionName must be a literal string", "description": "UiWithScreen/UiDoAction/UiAssert steps must set uiConnectionName to a literal string (the resultName from UiConnect/ApexConnect). Using for uiConnectionName causes Provar to error.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Set uiConnectionName using AdminConnection (not a variable reference).", @@ -1309,9 +1158,7 @@ "category": "TestCaseDesign", "name": "SOQL queries have SELECT and FROM clauses", "description": "Valid SOQL queries must contain both SELECT and FROM clauses.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Ensure SOQL query has format: SELECT FROM ", @@ -1327,9 +1174,7 @@ "category": "TestCaseDesign", "name": "SOQL queries include WHERE or LIMIT clause", "description": "SOQL queries must have WHERE clause or LIMIT to prevent unreasonably large result sets and potential heap size errors. This prevents Apex governance violations and performance issues.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Add WHERE clause to filter results (e.g. WHERE Id = :recordId) or LIMIT clause to cap result size (e.g. LIMIT 100). For large data volumes, always include both WHERE and LIMIT.", @@ -1345,9 +1190,7 @@ "category": "TestCaseDesign", "name": "SOQL queries must not be inside loops", "description": "SOQL queries inside For/ForEach/While/DoWhile loops violate Apex governor limits. Each SOQL query execution consumes from the 100 SOQL query limit per transaction. Queries in loops can quickly exceed this limit causing 'Too many SOQL queries: 101' errors.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Move SOQL query outside the loop and store results in a collection. Then iterate over the collection. Use SOQL WHERE clauses to filter in the query rather than filtering in a loop. Consider using aggregate queries or bulk processing patterns.", @@ -1363,9 +1206,7 @@ "category": "ApexAPI", "name": "Apex API steps must reference a valid connection", "description": "All Apex API calls (ApexCreate, ApexUpdate, ApexDelete, ApexRead, ApexSoqlQuery) must reference a valid Salesforce connection defined earlier in the test using ApexConnect. Tests will fail if connection is missing, empty, or references an undefined connection name.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Add an ApexConnect step before any Apex API calls to establish a Salesforce connection. Reference the connection name in all subsequent Apex API calls. Use variables for connection names if dynamically determined.", @@ -1380,9 +1221,7 @@ "category": "ApexAPI", "name": "Apex CRUD operations must have valid object types", "description": "ApexCreate, ApexUpdate, ApexDelete, and ApexRead steps must specify a valid Salesforce object type (e.g., Account, Contact, Broker__c). Object type must start with capital letter and contain only alphanumeric characters and underscores. Invalid or missing object types cause test failures.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Specify the correct Salesforce object API name (e.g., Account, Contact, Custom__c). Use standard object names or custom object API names ending with __c. Verify object exists in the target Salesforce org.", @@ -1397,9 +1236,7 @@ "category": "ApexAPI", "name": "ApexUpdateObject must have valid record ID", "description": "ApexUpdateObject requires a valid Salesforce record ID (15 or 18 characters) or a variable reference containing the ID. Missing, empty, or invalid record IDs cause test failures. Record IDs must be alphanumeric Salesforce IDs.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Provide a valid 15 or 18 character Salesforce record ID, or use a variable like {AccountId} that contains a valid ID from a prior Create or Read operation. Verify the record exists before attempting update.", @@ -1414,9 +1251,7 @@ "category": "ApexAPI", "name": "ApexUpdateObject must specify fields to update", "description": "ApexUpdateObject steps should specify at least one field to update in the field definitions. Update operations without field specifications have no effect and represent incomplete test logic. System arguments (objectType, recordId, objectId, apexConnectionName, resultScope, assignmentRuleId, resultIdName) are excluded from this check.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Add field definitions to the ApexUpdateObject step and provide values for at least one field to update. For example, specify Name, Status, or other relevant fields with test data or variable references.", @@ -1431,9 +1266,7 @@ "category": "ApexAPI", "name": "ApexDeleteObject must have valid record ID", "description": "ApexDeleteObject requires a valid Salesforce record ID (15 or 18 characters) or a variable reference containing the ID. Missing, empty, or invalid record IDs cause test failures. Record IDs must be alphanumeric Salesforce IDs.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Provide a valid 15 or 18 character Salesforce record ID, or use a variable like {AccountId} that contains a valid ID from a prior Create or Read operation. Verify the record exists before attempting delete.", @@ -1448,9 +1281,7 @@ "category": "ApexAPI", "name": "ApexReadObject must have valid record ID", "description": "ApexReadObject requires a valid Salesforce record ID (15 or 18 characters) or a variable reference containing the ID. Missing, empty, or invalid record IDs cause test failures. Record IDs must be alphanumeric Salesforce IDs.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Provide a valid 15 or 18 character Salesforce record ID, or use a variable like {AccountId} that contains a valid ID from a prior Create operation. Verify the record exists before attempting read.", @@ -1465,9 +1296,7 @@ "category": "ApexAPI", "name": "ApexCreateObject with fields must populate at least one field", "description": "When ApexCreateObject includes a fields structure, at least one field must be populated with a value. Empty field structures indicate incomplete test implementation and may cause test failures or create records with default values only.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Populate at least one field with a value or variable reference in the ApexCreateObject fields structure. Remove the fields structure if not needed. Verify required fields for the object are populated.", @@ -1482,9 +1311,7 @@ "category": "ApexAPI", "name": "ApexCreateObject and ApexUpdateObject must include parameter metadata", "description": "ApexCreateObject and ApexUpdateObject steps MUST include and sections after . These metadata sections enable Provar IDE features like field auto-complete, field suggestions, and proper object binding. Missing metadata causes degraded IDE experience and prevents field discovery.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Add with ConnectionName and CustomObjectName properties, and with apiParam elements for each field used in arguments. Study existing tier4 examples to see the exact structure required.", @@ -1499,9 +1326,7 @@ "category": "ApexAPI", "name": "ApexUpdateObject/ApexCreateObject field arguments must be direct, not nested in uiObjectFieldValue", "description": "ApexUpdateObject and ApexCreateObject steps MUST use direct field arguments (e.g., ) instead of nesting fields within uiObjectFieldValue elements inside the values argument. The values argument should contain only an empty valueList. Nested uiObjectFieldValue structures cause runtime failures and prevent proper field serialization.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Use direct field arguments: FieldValue with a separate empty . Do NOT nest fields inside uiObjectFieldValue within the values argument.", @@ -1516,9 +1341,7 @@ "category": "ApexAPI", "name": "ApexReadObject must use generatedParameters for field references, not fields argument with textType", "description": "ApexReadObject steps should NOT use a 'fields' argument containing textType elements for field names. Instead, fields must be properly referenced via generatedParameters with modelBinding attributes. The incorrect pattern uses FieldName which prevents proper field metadata and IDE integration. The correct pattern includes proper with elements.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Remove the 'fields' argument with textType elements. Ensure each field has a corresponding entry in with proper modelBinding. Fields should be declared through generatedParameters, not through a fields argument.", @@ -1533,9 +1356,7 @@ "category": "XMLSchema", "name": "Apex CRUD apiParam elements must be self-closing without summary/type children", "description": "In ApexCreateObject, ApexUpdateObject, and ApexReadObject steps, elements within MUST be self-closing tags without child elements. LLMs commonly hallucinate and elements inside apiParam, which causes Provar to reject the test case as invalid XML. The correct pattern is: . The hallucinated pattern includes description and ... children which are ONLY valid in other contexts (RestRequest, UiDoAction) but NEVER in Apex CRUD operations.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 10, "recommendation": "Remove all and child elements from apiParam elements in ApexCreateObject, ApexUpdateObject, and ApexReadObject steps. Use self-closing tags. For example: ", @@ -1551,9 +1372,7 @@ "category": "ApexAPI", "name": "ApexReadObject should use resultAssertions instead of separate AssertValues", "description": "When asserting field values after ApexReadObject, use within the ApexReadObject step rather than separate AssertValues API calls. This pattern is more efficient, provides better error messages, and maintains field metadata associations. The resultAssertions element should contain children with comparisonType, resultName, title, and optional elements.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "minor", "weight": 3, "recommendation": "Add block within ApexReadObject after the section. Move AssertValues logic into elements with appropriate comparisonType (EqualTo, IsPresent, NotEqualTo, etc.) and expected values where applicable.", @@ -1568,9 +1387,7 @@ "category": "ApexAPI", "name": "ApexExtractLayout must have object, file type, and path", "description": "ApexExtractLayout requires three parameters: object (Salesforce object API name), fileType (.xlsx only supported), and filePath (where to save the layout). Missing any parameter causes test failure. Only .xlsx file format is supported for layout extraction.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Specify the Salesforce object API name, set fileType to .xlsx, and provide a valid file path for the extracted layout. Ensure the directory path exists and is writable. Use relative paths for portability.", @@ -1585,9 +1402,7 @@ "category": "ApexAPI", "name": "ApexAssertLayout must have object and expected file", "description": "ApexAssertLayout requires two parameters: object (Salesforce object API name) and expectedFile (path to expected .xlsx layout file). Missing either parameter causes test failure. Only .xlsx file format is supported for layout assertions.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Specify the Salesforce object API name and provide path to expected .xlsx layout file. Ensure the expected file exists and is in .xlsx format. Use ApexExtractLayout to generate baseline layouts.", @@ -1602,9 +1417,7 @@ "category": "ApexAPI", "name": "Connection arguments must use correct naming convention", "description": "Different step types require specific connection argument names: 'connectionName' for Connect steps (ApexConnect, UiConnect, DbConnect, WebConnect) and web requests (RestRequest, SoapRequest); 'apexConnectionName' for Apex CRUD/SOQL operations (ApexCreateObject, ApexReadObject, ApexUpdateObject, ApexDeleteObject, ApexSoqlQuery, ApexBulk, etc.); 'uiConnectionName' for UI steps (UiWithScreen, UiDoAction, UiAssert); 'dbConnectionName' for database operations (DbDelete, DbInsert, DbRead, DbUpdate, SqlQuery). Using incorrect connection argument names causes connection lookup failures.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "minor", "weight": 2, "recommendation": "Use the correct connection argument name for each step type. Connect steps and web requests use 'connectionName'. Apex CRUD/SOQL operations use 'apexConnectionName'. UI steps use 'uiConnectionName'. Database operations use 'dbConnectionName'. Review Provar API documentation for the specific step type being used.", @@ -1619,9 +1432,7 @@ "category": "ApexAPI", "name": "Apex CRUD operations should include parameterGeneratorUri", "description": "ApexCreateObject, ApexReadObject, and ApexUpdateObject steps should include the parameterGeneratorUri attribute for proper IDE support. This attribute enables field auto-complete, field suggestions, and proper object binding in Provar IDE. Missing this attribute degrades IDE experience and may cause issues in some Provar versions.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "minor", "weight": 2, "recommendation": "Add parameterGeneratorUri attribute to Apex CRUD steps: ApexCreateObject uses 'command:com.provar.plugins.forcedotcom.ui.commands.CreateCustomObjectTestStepCommand', ApexReadObject uses 'command:com.provar.plugins.forcedotcom.ui.commands.ReadCustomObjectTestStepCommand', ApexUpdateObject uses 'command:com.provar.plugins.forcedotcom.ui.commands.UpdateCustomObjectTestStepCommand'.", @@ -1636,9 +1447,7 @@ "category": "TestCaseDesign", "name": "AssertValues arguments must be in correct order", "description": "AssertValues steps should contain the variable being tested (from Read/Query operations) and actualValue containing the expected literal value. Reversed arguments cause assertion logic to be backwards - comparing literals against variables instead of variables against expected values. This can result in unclear test step outcomes, although results are the same.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "info", "weight": 1, "recommendation": "Correct the argument order: expectedValue should contain the variable reference (e.g., {OpportunityRead.Name}), actualValue should contain the expected literal value (e.g., 'Demo Opportunity'). The assertion logic is: 'Assert that expectedValue equals actualValue', which reads as 'Assert that {variable} equals literal'.", @@ -1653,9 +1462,7 @@ "category": "TestCaseDesign", "name": "Must use AssertValues API, not deprecated Assert API", "description": "Test cases must use the supported com.provar.plugins.bundled.apis.AssertValues API with expectedValue/actualValue/comparisonType arguments. The com.provar.plugins.bundled.core.testapis.Assert API is not supported and will not load properly in Provar. AssertValues provides more flexible comparison operators and clearer assertion semantics.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Replace com.provar.plugins.bundled.core.testapis.Assert steps with com.provar.plugins.bundled.apis.AssertValues. Use expectedValue (the variable being tested), comparisonType (EqualTo, NotEqualTo, etc.), actualValue (the expected result), and failureMessage (explanation of what failed).", @@ -1670,9 +1477,7 @@ "category": "TestCaseDesign", "name": "AssertValues should have meaningful expected values", "description": "AssertValues steps using comparison operators (EqualTo, Contains, GreaterThan, etc.) should specify meaningful expectedValue arguments. Empty or missing expected values defeat the purpose of assertions and provide no validation. Without expected values, assertions cannot verify correct behavior. Comparison types requiring values: EqualTo, NotEqualTo, Contains, NotContains, StartsWith, NotStartsWith, EndsWith, NotEndsWith, Matches, NotMatches, GreaterThan, LessThan, GreaterThanOrEqualTo, LessThanOrEqualTo.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Specify a clear expectedValue for the assertion based on requirements. Use test data variables or literals that represent the expected state. Empty expected values provide no validation. For example: expectedValue='{Account.Name}', comparisonType='EqualTo', actualValue='ACME Corporation'.", @@ -1681,16 +1486,14 @@ "target": "step", "apiId": "com.provar.plugins.bundled.apis.AssertValues" }, - "source": "Provar Test Step Schema: Meaningful Assertions (Phase 2 API Usage Analysis)" + "source": "Provar Test Step Schema: Meaningful Assertions (API Usage Analysis)" }, { "id": "CONTROL-SLEEP-001", "category": "TestCaseDesign", "name": "Sleep step duration and frequency issues", "description": "Sleep steps with duration > 10 seconds cause unnecessary test execution time (major). Sleep inside loops multiplies wait time and indicates polling anti-pattern (minor). More than 5 sleeps per 50 steps indicates excessive waiting rather than proper synchronization (minor).", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Replace long sleeps with explicit waits for conditions. Move sleeps outside loops and use proper wait-until-ready patterns. Replace multiple sleeps with single targeted waits. Use UI wait conditions, API polling with timeout, or event-driven synchronization.", @@ -1705,9 +1508,7 @@ "category": "TestCaseDesign", "name": "While loop must have termination condition", "description": "While loops without counterEnd and without a terminating condition (<=, <, >=, >, ==, !=) can run infinitely if no termination is defined (major). While loops with condition='true' or '{true}' are infinite loops that rely solely on break statements, making them difficult to debug (minor). A loop is valid if it has either counterEnd OR a comparison condition.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Either set counterEnd to a reasonable limit (e.g., 10, 50, 100) OR use a terminating condition with a comparison operator (e.g., {Counter <= MaxIterations}). Avoid while{true} - use a meaningful condition instead. Add timeout handling and clear exit conditions. Consider using For loop with fixed iteration count if possible.", @@ -1722,9 +1523,7 @@ "category": "TestCaseDesign", "name": "Finally block must have description and be at end", "description": "Finally blocks without descriptions make it unclear what cleanup actions are performed (major). Finally blocks not at the end of the test may not execute if test stops before reaching them (minor). Finally is meant for cleanup that always runs regardless of test outcome.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Add clear description to Finally block explaining what cleanup actions are performed (e.g., 'Delete test data', 'Logout'). Move Finally block to the end of the test to ensure cleanup runs. Use Finally for critical cleanup like deleting test records or closing connections.", @@ -1739,9 +1538,7 @@ "category": "TestCaseDesign", "name": "SOQL queries specify resultListName", "description": "SOQL queries must specify resultListName to capture query results.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add 'resultListName' argument to store query results.", @@ -1757,9 +1554,7 @@ "category": "TestCaseDesign", "name": "Boolean values are 'true' or 'false'", "description": "Boolean value elements must contain exactly 'true' or 'false' (lowercase).", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Set boolean value to 'true' or 'false' (not 'True', 'FALSE', '1', '0', etc.).", @@ -1774,9 +1569,7 @@ "category": "TestCaseDesign", "name": "Numeric values are valid numbers", "description": "DISABLED: Rule generates false positives. Flags numeric-looking strings in SetValues without checking Salesforce field types. Many text fields (ZIP codes, IDs with leading zeros, year fields) legitimately store numeric strings.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "info", "weight": 0, "recommendation": "Review if this value should be valueClass='decimal' for calculations, or valueClass='string' for text fields like IDs, ZIP codes, phone numbers. Consider field type in Salesforce.", @@ -1791,9 +1584,7 @@ "category": "TestCaseDesign", "name": "SetValues steps have namedValues container", "description": "SetValues steps must have container element.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add container inside values argument.", @@ -1809,9 +1600,7 @@ "category": "NamingConventions", "name": "SetValues namedValue elements have name attribute", "description": "Each namedValue element must have a 'name' attribute defining the variable name.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add 'name' attribute to each element.", @@ -1827,9 +1616,7 @@ "category": "TestCaseDesign", "name": "SetValues namedValue elements have value element", "description": "Each namedValue must contain a element defining the variable value.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add element inside each .", @@ -1845,9 +1632,7 @@ "category": "StructureAndGrouping", "name": "SetValues must not contain invalid child elements", "description": "SetValues steps must use the correct structure with containing elements. AI models sometimes hallucinate invalid elements like , , or nested incorrectly. The correct structure is: .........", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 10, "recommendation": "Use correct SetValues structure: VariableName...Test. Each variable requires one block with three children: valuePath, value, and valueScope.", @@ -1865,9 +1650,7 @@ "category": "TestCaseDesign", "name": "Variable property references must be valid", "description": "Variable property references () must reference valid properties. For SOQL query results, only fields in the SELECT clause are valid. Common hallucinations include: .size, .length, .count (use Count() function), duplicate path elements, and referencing fields not in the query.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 6, "recommendation": "For list size, use Count(variable) function. For properties, ensure the field is included in the SOQL SELECT clause or ApexReadObject generatedParameters. Remove duplicate path elements.", @@ -1884,9 +1667,7 @@ "category": "TestCaseDesign", "name": "Manual cleanup matches object creation", "description": "When autoCleanup=false, test should have ApexDeleteObject steps matching ApexCreateObject steps.", - "appliesTo": [ - "TestCase" - ], + "appliesTo": ["TestCase"], "severity": "major", "weight": 5, "recommendation": "Add ApexDeleteObject steps for created objects or set autoCleanup=true.", @@ -1901,9 +1682,7 @@ "category": "TestCaseDesign", "name": "Cleanup deletes objects in reverse order", "description": "When manually deleting objects, delete child objects before parent objects.", - "appliesTo": [ - "TestCase" - ], + "appliesTo": ["TestCase"], "severity": "minor", "weight": 2, "recommendation": "Reorder ApexDeleteObject steps: delete child records before parent records.", @@ -1918,9 +1697,7 @@ "category": "TestCaseDesign", "name": "ApexCreateObject steps specify resultIdName", "description": "ApexCreateObject should specify resultIdName to capture created record ID for later reference or cleanup.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Add 'resultIdName' argument to store created record ID (e.g. 'CreatedAccountId', 'OpportunityId').", @@ -1936,9 +1713,7 @@ "category": "ReusabilityAndCallables", "name": "Called test cases are marked as Callable", "description": "Test cases invoked via TestCaseCall must have visibility='Callable' attribute.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Set visibility='Callable' on called test case or reference a callable test.", @@ -1954,9 +1729,7 @@ "category": "TestCaseDesign", "name": "Log messages use appropriate log levels", "description": "Log steps should use appropriate log levels: INFO for general info, WARN for warnings, ERROR for errors.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "info", "weight": 1, "recommendation": "Set logLevel argument to appropriate value: INFO, WARN, ERROR, or DEBUG.", @@ -1972,9 +1745,7 @@ "category": "TestCaseDesign", "name": "UiAssert steps specify assertion type", "description": "UiAssert steps should specify assertionType (Visible, Value, Enabled, etc.) for clarity.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "minor", "weight": 2, "recommendation": "Add 'assertionType' argument specifying the type of assertion.", @@ -1990,9 +1761,7 @@ "category": "TestCaseDesign", "name": "UiDoAction Set requires value argument", "description": "UiDoAction steps with interactionType='Set' must have a value argument.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add 'value' argument when using Set interaction type.", @@ -2008,9 +1777,7 @@ "category": "TestCaseDesign", "name": "Ui locator built-in actions use object binding", "description": "For Salesforce built-in actions (New, Edit, Save, Convert, etc.), uiLocator URIs must use an object binding (sf:ui:binding:object?object=...&action=...). Using the action binding form (sf:ui:binding:action?actionName=...) is known to cause runtime execution failures in generated tests.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Use an object-binding locator like: ui:locator?name=&binding=sf%3Aui%3Abinding%3Aobject%3Fobject%3D%26action%3D (replace object/action accordingly).", @@ -2026,9 +1793,7 @@ "category": "TestCaseDesign", "name": "UiAssert fieldLocator uses object+field binding", "description": "UiAssert embedded field assertions (uiFieldAssertion) must use a fieldLocator with an object binding (sf:ui:binding:object?field=&object=) and a locator name matching the field (name=). Incorrect fieldLocator bindings can cause runtime failures or assertions targeting the wrong element.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Use one of: (1) sf:ui:locator?name=&binding=sf%3Aui%3Abinding%3Aobject%3Ffield%3D%26object%3D (ensure name and field match), (2) ui:pageobject:field?field=&pageId=pageobjects., or (3) ui:locator?name= for Page Object locator references.", @@ -2044,9 +1809,7 @@ "category": "TestCaseDesign", "name": "UiAssert fieldAssertion must not wrap fieldLocator in uiLocator", "description": "UiAssert embedded field assertions (uiFieldAssertion) must use fieldLocator with a uri attribute directly. DO NOT wrap the locator in a nested element. The correct format is not . This incorrect nesting causes runtime failures.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Use directly without wrapping in .", @@ -2062,9 +1825,7 @@ "category": "TestCaseDesign", "name": "UiAssert bare locator in Salesforce metadata context causes render failure", "description": "UiAssert uiFieldAssertion using a bare locator reference (ui:locator?name=X without a binding parameter) inside a UiWithScreen whose target is a Salesforce metadata context (sf:ui:target?object=...) will fail to render in Provar Automation. Bare locators are only valid inside Page Object contexts. In Salesforce metadata contexts, the fieldLocator must include a full binding (ui:locator?name=X&binding=sf%3Aui%3Abinding%3Aobject%3Fobject%3D...%26field%3D...). This commonly occurs when asserting button visibility (e.g., Run, Edit, Delete buttons) as uiFieldAssertions — buttons are not Salesforce fields and cannot be asserted this way.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 10, "recommendation": "Remove the bare-locator uiFieldAssertion for non-field elements (buttons, actions). If you need to verify a button exists, use a UiDoAction click interaction instead of a UiAssert fieldAssertion. For Salesforce field assertions, always include the full binding: . NEVER use bare ui:locator?name=X without a binding inside a Salesforce metadata UiWithScreen.", @@ -2080,9 +1841,7 @@ "category": "TestCaseDesign", "name": "UiDoAction locator URIs must use valid patterns", "description": "Validates all UiDoAction locator URI patterns including: (1) SF locators: ui:locator?name=...&binding=sf:ui:binding:object?... with required name and binding params, (2) Page Object Fields: ui:pageobject:field?field=...&pageId=pageobjects... with required field and pageId params, (3) Page Object Methods: ui:pageobject:method?pageId=...&operation=... with required pageId and operation params, (4) Lead Conversion context-aware patterns (Convert on View vs submitConvert in Dialog). Detects malformed URIs like '%3Action' which should be '%3Faction'.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "SF locators: ui:locator?name=&binding=sf%3Aui%3Abinding%3Aobject%3F. Page Object Fields: ui:pageobject:field?field=&pageId=pageobjects.. Page Object Methods: ui:pageobject:method?pageId=pageobjects.&operation=. Ensure '%3F' (?) precedes query params in bindings, not '%3A' (colon).", @@ -2099,9 +1858,7 @@ "category": "TestCaseDesign", "name": "UiWithScreen target URIs must use valid patterns", "description": "Validates all UiWithScreen target URI patterns including: (1) SF targets with valid parameters (object/action, lightningComponent, lightningWebComponent, auraComponent, application/tab, lookup, fieldService, action-only), (2) Page Object targets: ui:pageobject:target?pageId=pageobjects... with required pageId param starting with 'pageobjects.' prefix.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "SF Object-Action targets: sf:ui:target?object=&action=. SF Lightning targets: sf:ui:target?lightningComponent=. Page Object targets: ui:pageobject:target?pageId=pageobjects..", @@ -2118,9 +1875,7 @@ "category": "TestCaseDesign", "name": "ApexConnect reuseConnectionName should be left blank", "description": "The reuseConnectionName argument in ApexConnect steps should be left empty/blank. AI models may hallucinate invalid values like 'Reuse' which cause runtime errors.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Leave reuseConnectionName blank: or . Do not use string values.", @@ -2137,9 +1892,7 @@ "category": "XMLSchema", "name": "ApexConnect - Only valid argument IDs allowed", "description": "ApexConnect steps must use ONLY the 21 valid argument IDs defined in Provar schema. AI models commonly hallucinate plausible-sounding but invalid arguments like autoPopulateRequiredFields, assertObjectFieldsPopulated, commandTimeout which cause Provar to reject the test case.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 10, "recommendation": "Only use these 21 valid ApexConnect argument IDs: connectionName, resultName, resultScope, uiApplicationName, quickUiLogin, closeAllPrimaryTabs, reuseConnectionName, alreadyOpenBehaviour, autoCleanup, cleanupConnectionName, logFileLocation, connectionId, enableObjectIdLogging, privateBrowsingMode, lightningMode, username, password, securityToken, environment, webBrowser. If you don't have a value, leave the argument empty: rather than inventing values.", @@ -2178,9 +1931,7 @@ "category": "XMLSchema", "name": "ApexConnect connectionId must use valueClass='id'", "description": "The connectionId argument in ApexConnect steps MUST use valueClass='id' with a GUID format value (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx), NOT valueClass='string'. Using valueClass='string' breaks connection references in Provar.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 10, "recommendation": "Use correct format: bce7fd3f-0f81-4c5c-ab68-c3edd44b5d1e. NEVER use valueClass=\"string\" or value=\"default\". If you don't have a specific GUID, leave the argument empty: ", @@ -2197,9 +1948,7 @@ "category": "TestCaseDesign", "name": "Wait arguments must use uiWait value class", "description": "The beforeWait, afterWait, and autoRetry arguments in UiDoAction and UiAssert steps must use class=\"uiWait\" with a uri attribute. Using valueClass=\"boolean\" is incorrect and causes runtime errors.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Use uiWait class with uri: or . Valid uri patterns: default, NoWait, ui:wait:timed?seconds=N, ui:wait:autoRetry:timeout=N, ui:wait:auraBusy?timeout=N, ui:wait:pageload?timeout=N.", @@ -2216,9 +1965,7 @@ "category": "TestCaseDesign", "name": "Save button locator must use correct pattern", "description": "Save button locators in UiDoAction steps must use lowercase 'save' and correct URI parameter order. Common error: using 'Save' (capital S) with action before object, or using %3A instead of %3F after object?", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Use correct pattern: name=save (lowercase), binding=sf%3Aui%3Abinding%3Aobject%3Fobject%3D%26action%3Dsave. Common mistakes: (1) Capital S in 'Save', (2) action before object in binding, (3) using %3Action instead of %3Faction, (4) using %3A instead of %3F after 'object?'", @@ -2235,9 +1982,7 @@ "category": "TestCaseDesign", "name": "UiWithScreen target uses invalid action value", "description": "The action parameter in sf:ui:target URIs must use valid Provar action names. AI models commonly hallucinate 'action=Home' when the correct value is 'action=ObjectHome'.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Use valid action values: ObjectHome (for object list/home), New, View, Edit, List, Convert, GlobalSearch, QuickAction, etc. Common fixes: Home→ObjectHome, Create→New, Update→Edit, Details→View.", @@ -2254,9 +1999,7 @@ "category": "TestCaseDesign", "name": "Sleep duration should be under 5 seconds", "description": "Fixed sleep durations should be kept short (<5 seconds). Prefer WaitFor with condition over fixed Sleep.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "info", "weight": 1, "recommendation": "Use WaitFor with condition instead of Sleep, or reduce sleep duration to <5 seconds.", @@ -2273,9 +2016,7 @@ "category": "StructureAndGrouping", "name": "Finally block should be at end of test", "description": "Finally blocks should be placed at the end of the test to ensure cleanup always executes.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Move Finally block to end of test case.", @@ -2291,9 +2032,7 @@ "category": "StructureAndGrouping", "name": "BDD scenario should start with Given", "description": "BDD test cases should start with Given step to establish preconditions.", - "appliesTo": [ - "TestCase" - ], + "appliesTo": ["TestCase"], "severity": "info", "weight": 1, "recommendation": "Start BDD scenario with Given step.", @@ -2308,9 +2047,7 @@ "category": "StructureAndGrouping", "name": "BDD steps should follow logical order", "description": "BDD steps should follow Given->When->Then order (with And/But as connectors).", - "appliesTo": [ - "TestCase" - ], + "appliesTo": ["TestCase"], "severity": "info", "weight": 1, "recommendation": "Reorder BDD steps to follow Given->When->Then pattern.", @@ -2325,9 +2062,7 @@ "category": "StructureAndGrouping", "name": "Limit And/But chain length", "description": "Chains of And/But steps should be limited to 3-5 for readability.", - "appliesTo": [ - "TestCase" - ], + "appliesTo": ["TestCase"], "severity": "info", "weight": 1, "recommendation": "Break long And/But chains into separate scenarios.", @@ -2343,9 +2078,7 @@ "category": "TestCaseDesign", "name": "ApexExecute code should be valid Apex syntax", "description": "Anonymous Apex code blocks should be syntactically valid.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Validate Apex syntax before execution. Test in Developer Console first.", @@ -2361,9 +2094,7 @@ "category": "TestCaseDesign", "name": "ApexBulk should be used for large data volumes", "description": "Use ApexBulk for operations with more than 200 records.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "info", "weight": 1, "recommendation": "Use ApexBulk instead of individual CRUD operations for >200 records.", @@ -2378,9 +2109,7 @@ "category": "TestCaseDesign", "name": "Prefer UiWithScreen over UiNavigate for Salesforce", "description": "For Salesforce navigation, prefer UiWithScreen over direct URL navigation with UiNavigate.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "info", "weight": 1, "recommendation": "Use UiWithScreen for Salesforce screens instead of UiNavigate.", @@ -2396,9 +2125,7 @@ "category": "TestCaseDesign", "name": "Verify fields after UiFill", "description": "UiFill should be followed by UiAssert to verify all fields were filled correctly.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "info", "weight": 1, "recommendation": "Add UiAssert steps after UiFill to verify field values.", @@ -2414,9 +2141,7 @@ "category": "TestCaseDesign", "name": "UiHandleAlert should capture alert text", "description": "When handling alerts with GetText action, store the text in a result variable.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "info", "weight": 1, "recommendation": "Set resultName when using GetText action on UiHandleAlert.", @@ -2432,9 +2157,7 @@ "category": "TestCaseDesign", "name": "Replace searchString should not be empty", "description": "Replace step searchString parameter should not be empty string.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Provide non-empty searchString for Replace operation.", @@ -2450,9 +2173,7 @@ "category": "TestCaseDesign", "name": "Split delimiter should not be empty", "description": "Split step delimiter parameter should not be empty string.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Provide non-empty delimiter for Split operation.", @@ -2468,9 +2189,7 @@ "category": "TestCaseDesign", "name": "Match regex pattern should be valid", "description": "When using isRegex=true, pattern must be valid regular expression.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Validate regex pattern syntax before using in Match step.", @@ -2486,19 +2205,14 @@ "category": "TestCaseDesign", "name": "DbDelete and DbUpdate should have WHERE clause", "description": "Database delete and update operations should include WHERE clause to avoid affecting all records.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add WHERE clause to DbDelete/DbUpdate to limit affected records.", "check": { "type": "dbWhereClause", "target": "step", - "apiIds": [ - "com.provar.plugins.bundled.apis.db.DbDelete", - "com.provar.plugins.bundled.apis.db.DbUpdate" - ] + "apiIds": ["com.provar.plugins.bundled.apis.db.DbDelete", "com.provar.plugins.bundled.apis.db.DbUpdate"] }, "source": "Database Testing Best Practices" }, @@ -2507,9 +2221,7 @@ "category": "TestCaseDesign", "name": "RestRequest method should be valid HTTP method", "description": "RestRequest method must be one of: GET, POST, PUT, DELETE, PATCH, HEAD, OPTIONS.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Use valid HTTP method (GET, POST, PUT, DELETE, PATCH).", @@ -2517,15 +2229,7 @@ "type": "restHttpMethod", "target": "step", "apiId": "com.provar.plugins.bundled.apis.restservice.RestRequest", - "validMethods": [ - "GET", - "POST", - "PUT", - "DELETE", - "PATCH", - "HEAD", - "OPTIONS" - ] + "validMethods": ["GET", "POST", "PUT", "DELETE", "PATCH", "HEAD", "OPTIONS"] }, "source": "REST API Testing Best Practices" }, @@ -2534,9 +2238,7 @@ "category": "TestCaseDesign", "name": "POST/PUT/PATCH should have request body", "description": "RestRequest with POST, PUT, or PATCH method should include request body.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Add request body for POST/PUT/PATCH operations.", @@ -2552,9 +2254,7 @@ "category": "TestCaseDesign", "name": "Validate REST response status", "description": "RestRequest should be followed by assertion to validate response status code.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "info", "weight": 1, "recommendation": "Add Assert step to validate REST response status code.", @@ -2570,9 +2270,7 @@ "category": "TestCaseDesign", "name": "SOAP request body should be well-formed XML", "description": "WebServiceRequest requestBody must be valid XML for SOAP operations.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Validate SOAP XML syntax before execution.", @@ -2588,9 +2286,7 @@ "category": "TestCaseDesign", "name": "AIAgentSession requires WebConnect first", "description": "AIAgentSession step must be preceded by WebConnect step.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add WebConnect step before AIAgentSession.", @@ -2606,9 +2302,7 @@ "category": "TestCaseDesign", "name": "AIAgentConversation requires valid session", "description": "AIAgentConversation must reference sessionID from AIAgentSession.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Ensure AIAgentSession is called first and sessionID is stored.", @@ -2624,9 +2318,7 @@ "category": "TestCaseDesign", "name": "GenerateUtterance count should be reasonable", "description": "GenerateUtterance count parameter should be between 1-100 for performance.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "info", "weight": 1, "recommendation": "Set GenerateUtterance count between 1-100.", @@ -2644,9 +2336,7 @@ "category": "TestCaseDesign", "name": "ImageValidator confidence should be 0.0-1.0", "description": "ImageValidator confidenceThreshold must be between 0.0 and 1.0.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Set confidenceThreshold between 0.0 and 1.0.", @@ -2664,9 +2354,7 @@ "category": "TestCaseDesign", "name": "ExtractSalesforceLayout before AssertSalesforceLayout", "description": "AssertSalesforceLayout should be preceded by ExtractSalesforceLayout to create baseline. Otherwise ensure you have a compatible data source to compare to.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "minor", "weight": 2, "recommendation": "Run ExtractSalesforceLayout first to create baseline layout file.", @@ -2682,9 +2370,7 @@ "category": "TestCaseDesign", "name": "ConvertLead status must be valid", "description": "ConvertLead convertedStatus must be a valid lead conversion status.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Use valid lead conversion status (Converted, Qualified, etc.).", @@ -2700,9 +2386,7 @@ "category": "TestCaseDesign", "name": "Read dataUrl should be valid file path", "description": "Read step dataUrl must point to existing Excel/CSV file.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Verify file exists before Read operation.", @@ -2718,9 +2402,7 @@ "category": "TestCaseDesign", "name": "Write dataUrl should be writable", "description": "Write step dataUrl directory must be writable.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Ensure output directory exists and is writable.", @@ -2736,9 +2418,7 @@ "category": "TestCaseDesign", "name": "Subscribe before ReceiveMessage", "description": "ReceiveMessage should be preceded by Subscribe to topic/channel.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Add Subscribe step before ReceiveMessage.", @@ -2754,9 +2434,7 @@ "category": "TestCaseDesign", "name": "ReceiveMessage timeout should be reasonable", "description": "ReceiveMessage timeout should be between 5-60 seconds.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "info", "weight": 1, "recommendation": "Set timeout between 5-60 seconds.", @@ -2774,10 +2452,7 @@ "category": "StructureAndGrouping", "name": "All arguments must have value elements", "description": "Every element must contain a child element, even if empty. Missing elements prevent test cases from loading in Provar.", - "appliesTo": [ - "TestCase", - "Step" - ], + "appliesTo": ["TestCase", "Step"], "severity": "critical", "weight": 10, "recommendation": "Add (or appropriate type) to all elements.", @@ -2793,10 +2468,7 @@ "category": "StructureAndGrouping", "name": "valueClass attributes must use lowercase", "description": "All valueClass attributes must use lowercase values (e.g., 'boolean' not 'Boolean', 'string' not 'String'). Incorrect casing prevents test cases from loading in Provar.", - "appliesTo": [ - "TestCase", - "Step" - ], + "appliesTo": ["TestCase", "Step"], "severity": "critical", "weight": 10, "recommendation": "Change all valueClass attributes to lowercase: boolean, string, decimal, integer, date, datetime, variable, compound, funcCall, value, valueList.", @@ -2811,10 +2483,7 @@ "category": "StructureAndGrouping", "name": "Boolean values must use lowercase", "description": "Boolean values in elements must be lowercase 'true' or 'false', not 'True', 'False', 'TRUE', or 'FALSE'. Incorrect casing prevents test cases from loading in Provar.", - "appliesTo": [ - "TestCase", - "Step" - ], + "appliesTo": ["TestCase", "Step"], "severity": "critical", "weight": 10, "recommendation": "Change all boolean values to lowercase: true, false.", @@ -2829,9 +2498,7 @@ "category": "StructureAndGrouping", "name": "Test case root element should not have unknown attributes", "description": "The root element should only contain known attributes: guid, id, name, visibility, registryId, failureBehaviour. Unknown attributes may prevent test cases from loading in Provar.", - "appliesTo": [ - "TestCase" - ], + "appliesTo": ["TestCase"], "severity": "minor", "weight": 3, "recommendation": "Remove unknown attributes from the root element.", @@ -2846,9 +2513,7 @@ "category": "TestCaseDesign", "name": "Test case should not be excessively long", "description": "Test cases with more than 150 steps become difficult to maintain, debug, and understand. Consider breaking long tests into smaller, focused test cases or using callable test cases.", - "appliesTo": [ - "TestCase" - ], + "appliesTo": ["TestCase"], "severity": "minor", "weight": 3, "recommendation": "Break test into smaller test cases or extract common logic into callable test cases.", @@ -2863,10 +2528,7 @@ "category": "TestCaseDesign", "name": "Picklist values should match Salesforce metadata", "description": "Picklist values used in test data should match the valid values defined in Salesforce. Common hallucinated values like 'Active' for Campaign.Status or 'Open' for Opportunity.StageName are often invalid and will cause test failures.", - "appliesTo": [ - "Step", - "TestCase" - ], + "appliesTo": ["Step", "TestCase"], "severity": "major", "weight": 7, "recommendation": "Use only picklist values provided in the metadata context. For Campaign.Status, valid values are typically: Planned, In Progress, Completed, Aborted. For Opportunity.StageName, check your org's sales process stages. For Lead.Status, check your org's lead process. Never assume generic values like 'Active', 'Inactive', 'Open', 'Closed' unless explicitly confirmed.", @@ -2890,17 +2552,43 @@ "category": "StructureAndGrouping", "name": "Value elements must use valid class attribute", "description": "The 'class' attribute on elements must be one of the valid Provar value classes. Invalid class values like 'null' cause runtime errors. Valid classes are: value, variable, compound, funcCall, valueList, uiWait, uiLocator, uiTarget, uiInteraction, restTarget, excelTarget, csvTarget, namedValues, url, template, and expression operators (add, sub, mult, div, eq, ne, gt, lt, ge, le, and, or, match).", - "appliesTo": [ - "Step", - "TestCase" - ], + "appliesTo": ["Step", "TestCase"], "severity": "critical", "weight": 10, "recommendation": "Use a valid class attribute. For literal values use class='value' with appropriate valueClass (string, boolean, decimal, id, date, dateTime). For variables use class='variable'. For empty/optional arguments, omit the element entirely: ", "check": { "type": "invalidValueClass", "target": "step", - "validClasses": ["value", "variable", "compound", "funcCall", "valueList", "uiWait", "uiLocator", "uiTarget", "uiInteraction", "restTarget", "excelTarget", "csvTarget", "namedValues", "url", "template", "add", "sub", "mult", "div", "eq", "ne", "gt", "lt", "ge", "le", "and", "or", "match"], + "validClasses": [ + "value", + "variable", + "compound", + "funcCall", + "valueList", + "uiWait", + "uiLocator", + "uiTarget", + "uiInteraction", + "restTarget", + "excelTarget", + "csvTarget", + "namedValues", + "url", + "template", + "add", + "sub", + "mult", + "div", + "eq", + "ne", + "gt", + "lt", + "ge", + "le", + "and", + "or", + "match" + ], "validValueClasses": ["string", "boolean", "decimal", "id", "date", "dateTime"] }, "notes": "Based on corpus analysis of 1451 test files with 329,424 elements. Common AI hallucination: class='null' (never valid). Empty arguments should have no child.", @@ -2911,16 +2599,22 @@ "category": "StructureAndGrouping", "name": "UiAssert steps must include all required arguments", "description": "UiAssert steps must include columnAssertions, pageAssertions, resultScope, captureAfter, beforeWait, and autoRetry arguments. Based on corpus analysis of 4,577 UiAssert instances, 100% include these arguments (even if empty). Missing these required arguments causes Provar validation failures at runtime.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 8, "recommendation": "Include all required arguments in UiAssert steps. Use empty value lists for unused assertions: . For resultScope use 'Test'. For captureAfter use 'false'. For beforeWait and autoRetry, include empty argument tags: ", "check": { "type": "uiAssertMissingArguments", "target": "step", - "requiredArguments": ["fieldAssertions", "columnAssertions", "pageAssertions", "resultScope", "captureAfter", "beforeWait", "autoRetry"], + "requiredArguments": [ + "fieldAssertions", + "columnAssertions", + "pageAssertions", + "resultScope", + "captureAfter", + "beforeWait", + "autoRetry" + ], "recommendedArguments": ["resultName"] }, "notes": "Based on corpus analysis of 4,577 UiAssert instances across 694 test files. All instances include these arguments. AI commonly omits columnAssertions, pageAssertions, resultScope, beforeWait, and autoRetry.", @@ -2931,9 +2625,7 @@ "category": "StructureAndGrouping", "name": "UiAssert steps must NOT contain generatedParameters", "description": "UiAssert steps should NOT contain element. Based on corpus analysis of 4,577 UiAssert instances, 0% contain generatedParameters. This element is 100% hallucinated by AI and causes Provar validation failures.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 10, "recommendation": "Remove the entire section from UiAssert steps. Unlike UiDoAction steps which use generatedParameters for field metadata, UiAssert steps never use this element. The assertion configuration is entirely within the arguments element.", @@ -2949,9 +2641,7 @@ "category": "LocatorPatterns", "name": "UI binding parameter order must have object= first", "description": "In UI binding URIs, the object= parameter MUST come FIRST, followed by field= or action=. Wrong order causes 'Unknown control' errors in Provar at runtime.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "critical", "weight": 10, "recommendation": "Correct the binding parameter order. Use 'object?object=ObjectName&action=ActionName' or 'object?object=ObjectName&field=FieldName'. Never put action= or field= before object=.", @@ -2979,9 +2669,7 @@ "category": "TestCaseDesign", "name": "UiDoAction/UiAssert fields should exist in Salesforce metadata", "description": "When Salesforce object metadata is provided in the context, UI operations (UiDoAction field sets, UiAssert field assertions) should only reference fields that exist in the metadata. Fields mentioned in user story but not present in Salesforce metadata may cause 'Unknown control' errors at runtime. This validation catches hallucinated or non-existent fields before test execution.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Only use fields that exist in the Salesforce org metadata. If a field from the user story shows 'Unknown control' in Provar, either: (1) the field doesn't exist in this org, (2) the field API name is different, or (3) the field isn't on the page layout. Check the org's metadata or remove the field operation from the test.", @@ -3001,9 +2689,7 @@ "category": "TestCaseDesign", "name": "Page Object locator references non-existent field", "description": "When Page Objects are provided, ui:pageobject:field locators must reference fields (WebElement variables) that actually exist in the corresponding Page Object Java class. AI models commonly hallucinate field names that are not defined in the provided Page Objects, causing 'Unknown control' errors at runtime.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 5, "recommendation": "Only reference field names that exist in the provided Page Object definitions. Check the Page Object class for available WebElement variables. If the field you need is not in the Page Object, either: (1) use SF metadata binding (sf:ui:binding:object) instead, or (2) ensure the Page Object includes the required field. Available fields are listed in the Page Object section of the generation context.", @@ -3023,9 +2709,7 @@ "category": "XMLSchema", "name": "funcCall id must be a valid Provar function", "description": "When using , the function name must be one of Provar's known built-in functions. Unknown function names cause runtime errors as Provar cannot evaluate them. Common hallucinations include JavaScript-style functions like 'concat', 'toString', 'parseInt' instead of Provar's actual functions.", - "appliesTo": [ - "TestCase" - ], + "appliesTo": ["TestCase"], "severity": "major", "weight": 6, "recommendation": "Use only valid Provar function names: Count, DateAdd, DateFormat, DateParse, GetEnvironmentVariable, GetSelectedEnvironment, IsSorted, Not, NumberFormat, Round, StringNormalize, StringReplace, StringTrim, TestCaseErrors, TestCaseName, TestCaseOutcome, TestCasePath, TestCaseSuccessful, TestRunErrors, UniqueId. NOTE: Concatenate, PadLeft, PadRight, Substring are NOT valid - use valueList for string building.", @@ -3057,9 +2741,7 @@ "category": "TestCaseDesign", "name": "UiAssert must use compound fields for component field assertions", "description": "Salesforce compound fields (Name=FirstName+LastName, BillingAddress=BillingStreet+BillingCity+BillingState+BillingPostalCode+BillingCountry, ShippingAddress, MailingAddress) are displayed as single fields in the UI View screen but set as individual component fields in UiDoAction. UiAssert steps MUST assert the compound field (e.g., 'Name', 'BillingAddress') using a compound value that combines the individual component variables, NOT assert the individual component fields directly which don't exist in the View UI.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 6, "recommendation": "For Name field: Assert 'Name' using . For Address fields: Assert 'BillingAddress' or 'ShippingAddress' as the compound field, not individual BillingStreet/BillingCity etc. The fieldLocator should reference the compound field name (e.g., field=Name, field=BillingAddress).", @@ -3069,9 +2751,39 @@ "apiId": "com.provar.plugins.forcedotcom.core.ui.UiAssert", "compoundFields": { "Name": ["FirstName", "LastName", "Salutation", "MiddleName", "Suffix"], - "BillingAddress": ["BillingStreet", "BillingCity", "BillingState", "BillingStateCode", "BillingPostalCode", "BillingCountry", "BillingCountryCode", "BillingLatitude", "BillingLongitude"], - "ShippingAddress": ["ShippingStreet", "ShippingCity", "ShippingState", "ShippingStateCode", "ShippingPostalCode", "ShippingCountry", "ShippingCountryCode", "ShippingLatitude", "ShippingLongitude"], - "MailingAddress": ["MailingStreet", "MailingCity", "MailingState", "MailingStateCode", "MailingPostalCode", "MailingCountry", "MailingCountryCode", "MailingLatitude", "MailingLongitude"] + "BillingAddress": [ + "BillingStreet", + "BillingCity", + "BillingState", + "BillingStateCode", + "BillingPostalCode", + "BillingCountry", + "BillingCountryCode", + "BillingLatitude", + "BillingLongitude" + ], + "ShippingAddress": [ + "ShippingStreet", + "ShippingCity", + "ShippingState", + "ShippingStateCode", + "ShippingPostalCode", + "ShippingCountry", + "ShippingCountryCode", + "ShippingLatitude", + "ShippingLongitude" + ], + "MailingAddress": [ + "MailingStreet", + "MailingCity", + "MailingState", + "MailingStateCode", + "MailingPostalCode", + "MailingCountry", + "MailingCountryCode", + "MailingLatitude", + "MailingLongitude" + ] } }, "examples": { @@ -3092,9 +2804,7 @@ "category": "TestCaseDesign", "name": "UiDoAction lookup fields should use Name values, not IDs", "description": "When setting lookup/reference fields via UiDoAction (UI automation), use the record Name (text value the user sees) instead of the record Id. The Salesforce UI expects display values like 'ABC Corp' for Account lookups, not 18-character IDs like '001xx000003DGWaAAO'. Using IDs in UI fields causes lookup failures. Note: Apex CRUD operations (ApexCreateObject, ApexUpdateObject) correctly use IDs for lookup fields.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "major", "weight": 6, "recommendation": "For UiDoAction on lookup fields (AccountId, ContactId, OwnerId, etc.): 1) Create a variable for the record Name BEFORE creating the record (e.g., AccountName = 'ABC Corp'), 2) Use that Name variable in UiDoAction for lookups, 3) Only use ID variables for Apex CRUD operations or sfUiTargetObjectId navigation. Example: Set AccountName='ABC Corp' via SetValues, then use {AccountName} for UiDoAction on AccountId field.", @@ -3124,20 +2834,25 @@ "category": "TestCaseDesign", "name": "Date/DateTime assertions should use proper format functions", "description": "Salesforce Date and DateTime fields are stored in specific formats: DateTime uses ISO 8601 format (2026-01-14T05:36:12.000+0000) and Date uses yyyy-MM-dd format (2026-01-14). When asserting these fields in UI or API tests, raw date strings may cause assertion failures due to format mismatches, timezone differences, or precision issues. Use Provar's date functions (DateFormat, DateParse, DateAdd, TODAY, NOW) for reliable date comparisons.", - "appliesTo": [ - "Step" - ], + "appliesTo": ["Step"], "severity": "minor", "weight": 4, "recommendation": "For date assertions: 1) Use DateFormat() to format dates consistently: DateFormat(variable, 'yyyy-MM-dd', 'GMT'), 2) Use DateParse() to parse date strings with known formats, 3) Use TODAY for current date comparisons (yyyy-MM-dd format), 4) Use NOW for current datetime (yyyy-MM-dd HH:mm:ss.SSS format), 5) Use DateAdd() for date calculations. Always specify timezone for DateTime comparisons. Example: Assert CloseDate equals DateFormat(ExpectedDate, 'yyyy-MM-dd').", "check": { "type": "dateFormatAssertion", "target": "step", - "apiIds": [ - "com.provar.plugins.forcedotcom.core.ui.UiAssert", - "com.provar.plugins.bundled.apis.AssertValues" + "apiIds": ["com.provar.plugins.forcedotcom.core.ui.UiAssert", "com.provar.plugins.bundled.apis.AssertValues"], + "dateFieldPatterns": [ + "Date$", + "DateTime$", + "^Close", + "^Start", + "^End", + "^Created", + "^LastModified", + "^Birth", + "^Expir" ], - "dateFieldPatterns": ["Date$", "DateTime$", "^Close", "^Start", "^End", "^Created", "^LastModified", "^Birth", "^Expir"], "sfDateTimeFormat": "yyyy-MM-dd'T'HH:mm:ss.SSSZ", "sfDateFormat": "yyyy-MM-dd" }, @@ -3162,10 +2877,7 @@ "category": "XMLSchema", "name": "valueClass='date' requires epoch timestamp, not date string", "description": "When using valueClass='date' or valueClass='dateTime' in Provar test cases, the value MUST be an epoch timestamp in milliseconds (a numeric value like 1736899200000), NOT an ISO date string like '2025-01-15' or '2025-01-15T00:00:00.000Z'. Using string date formats with valueClass='date' causes the test case to fail loading in Provar IDE. This is a critical XML schema violation that completely breaks the test case.", - "appliesTo": [ - "Step", - "TestCase" - ], + "appliesTo": ["Step", "TestCase"], "severity": "critical", "weight": 10, "recommendation": "Either: 1) Use valueClass='string' if you want to specify dates as strings like '2025-01-15', OR 2) Use valueClass='date' with epoch timestamp in milliseconds (e.g., 1736899200000 for 2025-01-15). For ApexCreateObject/ApexUpdateObject date fields, use valueClass='string' with the date in 'YYYY-MM-DD' format. To convert: new Date('2025-01-15').getTime() = 1736899200000.", @@ -3189,4 +2901,4 @@ "source": "Corpus analysis: All valueClass='date' values are epoch timestamps; string dates cause Provar IDE load failures" } ] -} \ No newline at end of file +} diff --git a/src/mcp/server.ts b/src/mcp/server.ts index 097bc5b..27b889f 100644 --- a/src/mcp/server.ts +++ b/src/mcp/server.ts @@ -32,6 +32,7 @@ import { registerAllAntTools } from './tools/antTools.js'; import { registerAllRcaTools } from './tools/rcaTools.js'; import { registerAllTestPlanTools } from './tools/testPlanTools.js'; import { registerAllNitroXTools } from './tools/nitroXTools.js'; +import { registerAllTestCaseStepTools } from './tools/testCaseStepTools.js'; import { registerAllConnectionTools } from './tools/connectionTools.js'; import { registerAllPrompts } from './prompts/index.js'; @@ -78,9 +79,10 @@ export function createProvarMcpServer(config: ServerConfig): McpServer { registerAllAutomationTools(server, config); registerAllDefectTools(server); registerAllAntTools(server, config); - registerAllRcaTools(server); + registerAllRcaTools(server, config); registerAllTestPlanTools(server, config); registerAllNitroXTools(server, config); + registerAllTestCaseStepTools(server, config); registerAllConnectionTools(server, config); // ── Provar prompts ─────────────────────────────────────────────────────────── diff --git a/src/mcp/tools/automationTools.ts b/src/mcp/tools/automationTools.ts index c09f27d..42b4038 100644 --- a/src/mcp/tools/automationTools.ts +++ b/src/mcp/tools/automationTools.ts @@ -109,7 +109,7 @@ function resolveSfExecutable(): string | null { // missing — it exits non-zero with "not recognised" in stderr but sets no // probe.error. Trying shell:false first catches both cases correctly. // - // Phase 1: shell:false (works on Linux/macOS; gives ENOENT on Windows if + // First attempt: shell:false (works on Linux/macOS; gives ENOENT on Windows if // sf.cmd is on PATH but requires the shell). const probe = sfSpawnHelper.spawnSync('sf', ['--version'], { encoding: 'utf-8', @@ -121,7 +121,7 @@ function resolveSfExecutable(): string | null { return cachedSfPath; } - // Phase 2 (Windows only): retry with shell:true when the plain probe failed + // Windows fallback: retry with shell:true when the plain probe failed // with ENOENT — meaning sf.cmd exists on PATH but can't run without the shell. if (platform === 'win32' && (probe.error as NodeJS.ErrnoException | undefined)?.code === 'ENOENT') { const probeShell = sfSpawnHelper.spawnSync('sf', ['--version'], { diff --git a/src/mcp/tools/rcaTools.ts b/src/mcp/tools/rcaTools.ts index 3202965..564b336 100644 --- a/src/mcp/tools/rcaTools.ts +++ b/src/mcp/tools/rcaTools.ts @@ -12,6 +12,8 @@ import path from 'node:path'; import { z } from 'zod'; import { XMLParser } from 'fast-xml-parser'; import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js'; +import type { ServerConfig } from '../server.js'; +import { assertPathAllowed, PathPolicyError } from '../security/pathPolicy.js'; import { makeError, makeRequestId } from '../schemas/common.js'; import { log } from '../logging/logger.js'; @@ -665,17 +667,23 @@ function buildFailureReports( // ── provar.testrun.rca tool ─────────────────────────────────────────────────── -export function registerTestRunRca(server: McpServer): void { +export function registerTestRunRca(server: McpServer, config: ServerConfig): void { server.tool( 'provar.testrun.rca', [ 'Parse a completed Provar test run and produce a structured Root Cause Analysis (RCA) report.', 'Resolves the results directory, parses JUnit.xml, classifies each failure by category,', 'and produces recommendations. Use locate_only=true to skip parsing and just resolve artifact locations.', + 'Use mode="failures" to get a lightweight array of failed test cases', + '([{ testItemId, title, errorMessage }]) without the full RCA classification — useful when you', + 'need failure names quickly without loading the HTML report.', ].join(' '), { project_path: z.string().describe('Absolute path to the Provar project root'), - results_path: z.string().optional().describe('Explicit override for the results base directory'), + results_path: z + .string() + .optional() + .describe('Explicit override for the results base directory; must be within --allowed-paths if provided'), run_index: z .number() .int() @@ -687,12 +695,33 @@ export function registerTestRunRca(server: McpServer): void { .optional() .default(false) .describe('If true, skip parsing and return just artifact locations'), + mode: z + .enum(['rca', 'failures']) + .optional() + .default('rca') + .describe( + '"rca" (default): full root-cause analysis with classification and recommendations. ' + + '"failures": lightweight array of failed test cases [{ testItemId, title, errorMessage }].' + ), }, (input) => { const requestId = makeRequestId(); - log('info', 'provar.testrun.rca', { requestId, locate_only: input.locate_only }); + log('info', 'provar.testrun.rca', { requestId, locate_only: input.locate_only, mode: input.mode }); try { + // ── Path policy ────────────────────────────────────────────────────── + const pathsToCheck: string[] = [path.resolve(input.project_path)]; + if (input.results_path) pathsToCheck.push(input.results_path); + for (const p of pathsToCheck) { + try { + assertPathAllowed(p, config.allowedPaths); + } catch (pErr: unknown) { + const e = pErr as Error & { code?: string }; + const err = makeError(e instanceof PathPolicyError ? e.code : 'PATH_NOT_ALLOWED', e.message, requestId); + return { isError: true, content: [{ type: 'text' as const, text: JSON.stringify(err) }] }; + } + } + // ── Resolve location ───────────────────────────────────────────────── const resolved = resolveResultsLocation(input.project_path, input.results_path, input.run_index); if ('error' in resolved) { @@ -741,6 +770,30 @@ export function registerTestRunRca(server: McpServer): void { }; } + // ── mode=failures: lightweight failure list ────────────────────────── + if (input.mode === 'failures') { + if (!junit_xml) { + const result = { + requestId, + results_dir, + failures: [] as Array<{ testItemId: string; title: string; errorMessage: string }>, + details: { warning: 'JUnit.xml not found in results directory — no failure data available' }, + }; + return { content: [{ type: 'text' as const, text: JSON.stringify(result) }] }; + } + const xmlContent = fs.readFileSync(junit_xml, 'utf-8'); + const parsed = parseJUnit(xmlContent); + const failures = parsed.testcases + .filter((tc) => tc.failureText !== null && !tc.isSkipped) + .map((tc) => ({ + testItemId: tc.name, + title: tc.name, + errorMessage: (tc.failureText ?? '').slice(0, 500), + })); + const result = { requestId, results_dir, failures }; + return { content: [{ type: 'text' as const, text: JSON.stringify(result) }] }; + } + // ── Check JUnit.xml exists ─────────────────────────────────────────── if (!junit_xml) { const result = { @@ -809,7 +862,7 @@ export function registerTestRunRca(server: McpServer): void { // ── Registration entry point ────────────────────────────────────────────────── -export function registerAllRcaTools(server: McpServer): void { +export function registerAllRcaTools(server: McpServer, config: ServerConfig): void { registerTestRunLocate(server); - registerTestRunRca(server); + registerTestRunRca(server, config); } diff --git a/src/mcp/tools/testCaseStepTools.ts b/src/mcp/tools/testCaseStepTools.ts new file mode 100644 index 0000000..7ba6e08 --- /dev/null +++ b/src/mcp/tools/testCaseStepTools.ts @@ -0,0 +1,267 @@ +/* + * Copyright (c) 2024 Provar Limited. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.md file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ + +/* eslint-disable camelcase */ +import fs from 'node:fs'; +import path from 'node:path'; +import { z } from 'zod'; +import { XMLParser, XMLBuilder } from 'fast-xml-parser'; +import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js'; +import type { ServerConfig } from '../server.js'; +import { assertPathAllowed, PathPolicyError } from '../security/pathPolicy.js'; +import { makeError, makeRequestId } from '../schemas/common.js'; +import { log } from '../logging/logger.js'; +import { validateTestCase } from './testCaseValidate.js'; + +// ── XML parse / build config ────────────────────────────────────────────────── + +const PARSER_OPTIONS = { + ignoreAttributes: false, + attributeNamePrefix: '@_', + parseAttributeValue: false, + isArray: (tagName: string): boolean => tagName === 'apiCall', +}; + +const BUILDER_OPTIONS = { + ignoreAttributes: false, + attributeNamePrefix: '@_', + format: true, + indentBy: ' ', + suppressEmptyNode: false, +}; + +type ApiCallNode = Record; + +// ── Helpers ─────────────────────────────────────────────────────────────────── + +function parseTestCaseXml(xmlContent: string): Record { + const parser = new XMLParser(PARSER_OPTIONS); + return parser.parse(xmlContent) as Record; +} + +function buildTestCaseXml(parsed: Record): string { + const builder = new XMLBuilder(BUILDER_OPTIONS); + return '\n' + (builder.build(parsed) as string); +} + +function getApiCalls(parsed: Record): ApiCallNode[] | null { + const tc = parsed['testCase']; + if (!tc || typeof tc !== 'object') return null; + const steps = (tc as Record)['steps']; + if (!steps || typeof steps !== 'object') return null; + const calls = (steps as Record)['apiCall']; + if (!Array.isArray(calls)) return null; + return calls as ApiCallNode[]; +} + +function collectAllTestItemIds(parsed: Record): string[] { + const calls = getApiCalls(parsed); + if (!calls) return []; + return calls.map((c) => c['@_testItemId']).filter((id): id is string => typeof id === 'string'); +} + +function parseNewStep(stepXml: string): { step: ApiCallNode } | { error: string } { + try { + const fragParser = new XMLParser(PARSER_OPTIONS); + const fragDoc = fragParser.parse(`${stepXml}`) as Record; + const rootEl = fragDoc['root'] as Record | undefined; + const callEl = rootEl?.['apiCall']; + if (!callEl) return { error: 'step_xml must contain exactly one element' }; + const calls = Array.isArray(callEl) ? (callEl as ApiCallNode[]) : [callEl as ApiCallNode]; + if (calls.length !== 1) return { error: 'step_xml must contain exactly one element' }; + return { step: calls[0] }; + } catch (e: unknown) { + return { error: (e as Error).message }; + } +} + +// ── Tool registration ───────────────────────────────────────────────────────── + +export function registerTestCaseStepEdit(server: McpServer, config: ServerConfig): void { + server.tool( + 'provar.testcase.step.edit', + [ + 'Add or remove a single step (apiCall) in a Provar XML test case file.', + 'Uses write-to-temp-then-rename to minimise partial-write risk.', + 'Prerequisites: the test case must exist and be valid XML.', + 'For mode=remove: supply test_item_id of the step to remove.', + 'For mode=add: supply test_item_id of the anchor step, position (before|after, default after),', + 'and step_xml (the ... XML fragment for the new step; must contain exactly one ).', + 'A backup is written to .bak before any mutation and restored automatically if', + 'the post-edit validation fails.', + 'Returns STEP_NOT_FOUND (with all_test_item_ids list) when the target step is absent.', + 'Returns INVALID_STEP_XML when step_xml cannot be parsed or contains ≠1 elements.', + 'Returns INVALID_XML_AFTER_EDIT (backup restored) when the mutated file fails validation.', + ].join(' '), + { + test_case_path: z.string().describe('Absolute path to the .testcase XML file; must be within --allowed-paths'), + mode: z.enum(['remove', 'add']).describe('"remove" to delete a step; "add" to insert a new step'), + test_item_id: z + .string() + .describe('For mode=remove: testItemId of the step to delete. For mode=add: testItemId of the anchor step.'), + position: z + .enum(['before', 'after']) + .optional() + .default('after') + .describe('Where to insert relative to the anchor step (mode=add only; default: after)'), + step_xml: z + .string() + .optional() + .describe( + 'The ... XML fragment for the new step (mode=add only). Must be well-formed XML.' + ), + validate_after_edit: z + .boolean() + .optional() + .default(true) + .describe('Run provar.testcase.validate after the mutation; restores backup on failure (default: true)'), + }, + (input) => { + const requestId = makeRequestId(); + log('info', 'provar.testcase.step.edit', { requestId, mode: input.mode, test_item_id: input.test_item_id }); + + try { + const resolvedPath = path.resolve(input.test_case_path); + const bakPath = resolvedPath + '.bak'; + + // Path policy — validate both the target file and its backup path + assertPathAllowed(resolvedPath, config.allowedPaths); + assertPathAllowed(bakPath, config.allowedPaths); + + // Validate step_xml up-front before touching the file + let newStep: ApiCallNode | null = null; + if (input.mode === 'add') { + if (!input.step_xml) { + const err = makeError('MISSING_INPUT', 'step_xml is required for mode=add', requestId); + return { isError: true, content: [{ type: 'text' as const, text: JSON.stringify(err) }] }; + } + const parsed_step = parseNewStep(input.step_xml); + if ('error' in parsed_step) { + const err = makeError('INVALID_STEP_XML', `step_xml parse error: ${parsed_step.error}`, requestId); + return { isError: true, content: [{ type: 'text' as const, text: JSON.stringify(err) }] }; + } + newStep = parsed_step.step; + } + + // Read the test case file + if (!fs.existsSync(resolvedPath)) { + const err = makeError('FILE_NOT_FOUND', `Test case not found: ${resolvedPath}`, requestId); + return { isError: true, content: [{ type: 'text' as const, text: JSON.stringify(err) }] }; + } + const original = fs.readFileSync(resolvedPath, 'utf-8'); + + // Parse + let parsed: Record; + try { + parsed = parseTestCaseXml(original); + } catch (e: unknown) { + const err = makeError('INVALID_XML', `Cannot parse test case: ${(e as Error).message}`, requestId); + return { isError: true, content: [{ type: 'text' as const, text: JSON.stringify(err) }] }; + } + + const apiCalls = getApiCalls(parsed); + if (!apiCalls) { + const err = makeError( + 'INVALID_XML', + 'Test case XML does not contain a structure', + requestId + ); + return { isError: true, content: [{ type: 'text' as const, text: JSON.stringify(err) }] }; + } + + // Find target step + const targetIndex = apiCalls.findIndex((c) => String(c['@_testItemId']) === input.test_item_id); + if (targetIndex === -1) { + const allIds = collectAllTestItemIds(parsed); + const err = makeError( + 'STEP_NOT_FOUND', + `Step with testItemId "${input.test_item_id}" not found in ${resolvedPath}`, + requestId, + false, + { all_test_item_ids: allIds } + ); + return { isError: true, content: [{ type: 'text' as const, text: JSON.stringify(err) }] }; + } + + // Mutate the parsed tree + if (input.mode === 'remove') { + apiCalls.splice(targetIndex, 1); + } else { + // mode=add + const insertAt = input.position === 'before' ? targetIndex : targetIndex + 1; + apiCalls.splice(insertAt, 0, newStep as ApiCallNode); + } + + // Rebuild XML + const mutatedXml = buildTestCaseXml(parsed); + + // Write backup, then write mutated file via temp→rename to minimise partial-write risk + const tmpPath = resolvedPath + '.tmp'; + fs.writeFileSync(bakPath, original, 'utf-8'); + fs.writeFileSync(tmpPath, mutatedXml, 'utf-8'); + fs.renameSync(tmpPath, resolvedPath); + + // Validate if requested + let validation: ReturnType | null | undefined; + if (input.validate_after_edit) { + try { + validation = validateTestCase(mutatedXml, path.basename(resolvedPath, '.testcase')); + } catch { + // treat thrown validation errors as failures + validation = null; + } + + if (!validation || !validation.is_valid) { + // Restore from backup + fs.writeFileSync(resolvedPath, original, 'utf-8'); + fs.unlinkSync(bakPath); + const err = makeError( + 'INVALID_XML_AFTER_EDIT', + `Validation failed after ${input.mode}; original file restored from backup`, + requestId, + false, + { validation_issues: validation?.issues ?? ['Validation threw an unexpected error'] } + ); + return { isError: true, content: [{ type: 'text' as const, text: JSON.stringify(err) }] }; + } + } + + // Success — delete backup + try { + fs.unlinkSync(bakPath); + } catch { + // non-fatal + } + + const result = { + requestId, + success: true, + test_item_id: input.test_item_id, + mode: input.mode, + ...(input.validate_after_edit && validation ? { validation } : {}), + }; + + return { + content: [{ type: 'text' as const, text: JSON.stringify(result) }], + }; + } catch (err: unknown) { + const error = err as Error & { code?: string }; + const errResult = makeError( + error instanceof PathPolicyError ? error.code : 'STEP_EDIT_ERROR', + error.message, + requestId + ); + log('error', 'provar.testcase.step.edit failed', { requestId, error: error.message }); + return { isError: true, content: [{ type: 'text' as const, text: JSON.stringify(errResult) }] }; + } + } + ); +} + +export function registerAllTestCaseStepTools(server: McpServer, config: ServerConfig): void { + registerTestCaseStepEdit(server, config); +} diff --git a/src/services/auth/credentials.ts b/src/services/auth/credentials.ts index e582f87..f8c9e5c 100644 --- a/src/services/auth/credentials.ts +++ b/src/services/auth/credentials.ts @@ -15,7 +15,7 @@ export interface StoredCredentials { prefix: string; set_at: string; source: 'manual' | 'cognito' | 'salesforce'; - // Phase 2 fields — optional so Phase 1 files remain valid after upgrade + // Extended fields — optional so earlier credential files remain valid after upgrade username?: string; tier?: string; expires_at?: string; diff --git a/test/unit/mcp/rcaTools.test.ts b/test/unit/mcp/rcaTools.test.ts index e419c0c..4f24e29 100644 --- a/test/unit/mcp/rcaTools.test.ts +++ b/test/unit/mcp/rcaTools.test.ts @@ -80,11 +80,13 @@ const UNKNOWN_JUNIT_XML = ` let tmpDir: string; let server: MockMcpServer; +const UNRESTRICTED_CONFIG = { allowedPaths: [] }; + beforeEach(() => { tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'rcatools-test-')); server = new MockMcpServer(); registerTestRunLocate(server as never); - registerTestRunRca(server as never); + registerTestRunRca(server as never, UNRESTRICTED_CONFIG); }); afterEach(() => { @@ -610,4 +612,104 @@ describe('provar.testrun.rca', () => { 'Salesforce categories should not appear in infrastructure_issues' ); }); + + it('mode=rca is the default and produces existing RCA output shape', () => { + const resultsDir = makeResultsDir(path.join(tmpDir, 'rca-default'), JUNIT_XML); + const body = parseText(server.call('provar.testrun.rca', { project_path: tmpDir, results_path: resultsDir })); + // Full RCA shape must be present + assert.ok('run_summary' in body, 'run_summary should be present for mode=rca'); + assert.ok('failures' in body, 'failures should be present'); + assert.ok('recommendations' in body, 'recommendations should be present'); + assert.ok('infrastructure_issues' in body, 'infrastructure_issues should be present'); + }); +}); + +// ── provar.testrun.rca mode=failures ────────────────────────────────────────── + +describe('provar.testrun.rca mode=failures', () => { + it('returns lightweight failure array when JUnit.xml is present', () => { + const resultsDir = makeResultsDir(path.join(tmpDir, 'failures-mode'), JUNIT_XML); + const body = parseText( + server.call('provar.testrun.rca', { project_path: tmpDir, results_path: resultsDir, mode: 'failures' }) + ); + + assert.ok('failures' in body, 'failures should be present'); + const failures = body['failures'] as Array>; + assert.equal(failures.length, 1, 'JUNIT_XML has exactly 1 failing test case'); + + const f = failures[0]; + assert.ok('testItemId' in f, 'testItemId should be present'); + assert.ok('title' in f, 'title should be present'); + assert.ok('errorMessage' in f, 'errorMessage should be present'); + assert.equal(f['testItemId'], 'SearchTest.testcase'); + assert.ok(typeof f['errorMessage'] === 'string' && f['errorMessage'].length > 0); + + // Should NOT contain full RCA fields + assert.ok(!('run_summary' in body), 'mode=failures should not include run_summary'); + assert.ok(!('infrastructure_issues' in body), 'mode=failures should not include infrastructure_issues'); + assert.ok(!('recommendations' in body), 'mode=failures should not include recommendations'); + }); + + it('returns empty array with warning when results dir has no JUnit.xml', () => { + const resultsDir = makeResultsDir(path.join(tmpDir, 'failures-empty')); + const body = parseText( + server.call('provar.testrun.rca', { project_path: tmpDir, results_path: resultsDir, mode: 'failures' }) + ); + + const failures = body['failures'] as unknown[]; + assert.equal(failures.length, 0, 'should return empty array'); + + const details = body['details'] as Record; + assert.ok(typeof details?.['warning'] === 'string', 'details.warning should be set'); + }); + + it('skipped tests are not included in failures list', () => { + const resultsDir = makeResultsDir(path.join(tmpDir, 'failures-skip'), JUNIT_XML); + const body = parseText( + server.call('provar.testrun.rca', { project_path: tmpDir, results_path: resultsDir, mode: 'failures' }) + ); + const failures = body['failures'] as Array>; + const testItemIds = failures.map((f) => f['testItemId']); + assert.ok(!testItemIds.includes('DataTest.testcase'), 'skipped test should not appear in failures'); + }); + + it('rejects explicit results_path outside allowed paths', () => { + const restrictedServer = new MockMcpServer(); + const allowedDir = path.join(tmpDir, 'allowed'); + fs.mkdirSync(allowedDir, { recursive: true }); + registerTestRunRca(restrictedServer as never, { allowedPaths: [allowedDir] }); + + const outsideDir = makeResultsDir(path.join(tmpDir, 'outside'), JUNIT_XML); + const result = restrictedServer.call('provar.testrun.rca', { + project_path: tmpDir, + results_path: outsideDir, + mode: 'failures', + }); + + assert.equal(isError(result), true); + const body = parseText(result); + assert.ok( + body['error_code'] === 'PATH_NOT_ALLOWED' || body['error_code'] === 'PATH_TRAVERSAL', + `expected path policy error, got: ${String(body['error_code'])}` + ); + }); + + it('rejects project_path outside allowed paths', () => { + const restrictedServer = new MockMcpServer(); + const allowedDir = path.join(tmpDir, 'allowed'); + fs.mkdirSync(allowedDir, { recursive: true }); + registerTestRunRca(restrictedServer as never, { allowedPaths: [allowedDir] }); + + const result = restrictedServer.call('provar.testrun.rca', { + project_path: tmpDir, // tmpDir root is outside allowed subdir + mode: 'failures', + }); + + assert.equal(isError(result), true); + const body = parseText(result); + assert.ok( + body['error_code'] === 'PATH_NOT_ALLOWED' || body['error_code'] === 'PATH_TRAVERSAL', + `expected path policy error for project_path, got: ${String(body['error_code'])}` + ); + }); }); diff --git a/test/unit/mcp/testCaseStepTools.test.ts b/test/unit/mcp/testCaseStepTools.test.ts new file mode 100644 index 0000000..af46cfa --- /dev/null +++ b/test/unit/mcp/testCaseStepTools.test.ts @@ -0,0 +1,312 @@ +/* + * Copyright (c) 2024 Provar Limited. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.md file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ + +/* eslint-disable camelcase */ +import { strict as assert } from 'node:assert'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { describe, it, beforeEach, afterEach } from 'mocha'; +import { registerAllTestCaseStepTools } from '../../../src/mcp/tools/testCaseStepTools.js'; + +// ── Mock MCP server ──────────────────────────────────────────────────────────── + +type ToolHandler = (args: Record) => unknown; + +class MockMcpServer { + private handlers = new Map(); + + public tool(name: string, _desc: string, _schema: unknown, handler: ToolHandler): void { + this.handlers.set(name, handler); + } + + public call(name: string, args: Record): ReturnType { + const h = this.handlers.get(name); + if (!h) throw new Error(`Tool not registered: ${name}`); + return h(args); + } +} + +// ── Helpers ──────────────────────────────────────────────────────────────────── + +function parseText(result: unknown): Record { + const r = result as { content: Array<{ type: string; text: string }> }; + return JSON.parse(r.content[0].text) as Record; +} + +function isError(result: unknown): boolean { + return (result as { isError?: boolean }).isError === true; +} + +// ── Fixture XML ──────────────────────────────────────────────────────────────── + +const VALID_TESTCASE_XML = ` + + + + + + +`; + +const NEW_STEP_XML = + ''; + +// ── Test setup ───────────────────────────────────────────────────────────────── + +let tmpDir: string; +let server: MockMcpServer; +let tcPath: string; + +beforeEach(() => { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'steptools-test-')); + server = new MockMcpServer(); + registerAllTestCaseStepTools(server as never, { allowedPaths: [] }); + tcPath = path.join(tmpDir, 'SmokeTest.testcase'); + fs.writeFileSync(tcPath, VALID_TESTCASE_XML, 'utf-8'); +}); + +afterEach(() => { + fs.rmSync(tmpDir, { recursive: true, force: true }); +}); + +// ── provar.testcase.step.edit ────────────────────────────────────────────────── + +describe('provar.testcase.step.edit', () => { + // remove happy path + it('mode=remove removes the target step and leaves file valid', () => { + const result = server.call('provar.testcase.step.edit', { + test_case_path: tcPath, + mode: 'remove', + test_item_id: '2', + validate_after_edit: false, + }); + + assert.equal(isError(result), false); + const body = parseText(result); + assert.equal(body['success'], true); + assert.equal(body['mode'], 'remove'); + assert.equal(body['test_item_id'], '2'); + + const written = fs.readFileSync(tcPath, 'utf-8'); + assert.ok(!written.includes('testItemId="2"'), 'step 2 should be removed'); + assert.ok(written.includes('testItemId="1"'), 'step 1 should remain'); + assert.ok(written.includes('testItemId="3"'), 'step 3 should remain'); + assert.ok(!fs.existsSync(tcPath + '.bak'), 'backup file should be deleted on success'); + }); + + // add happy path — after anchor + it('mode=add inserts new step after anchor by default', () => { + const result = server.call('provar.testcase.step.edit', { + test_case_path: tcPath, + mode: 'add', + test_item_id: '1', + step_xml: NEW_STEP_XML, + validate_after_edit: false, + }); + + assert.equal(isError(result), false); + const body = parseText(result); + assert.equal(body['success'], true); + assert.equal(body['mode'], 'add'); + + const written = fs.readFileSync(tcPath, 'utf-8'); + assert.ok(written.includes('testItemId="99"'), 'new step should be present'); + // Verify order: step 1 before step 99 + const pos1 = written.indexOf('testItemId="1"'); + const pos99 = written.indexOf('testItemId="99"'); + const pos2 = written.indexOf('testItemId="2"'); + assert.ok(pos1 < pos99, 'anchor step 1 should appear before new step 99'); + assert.ok(pos99 < pos2, 'new step 99 should appear before step 2'); + assert.ok(!fs.existsSync(tcPath + '.bak'), 'backup file should be deleted on success'); + }); + + // add before anchor + it('mode=add with position=before inserts before anchor', () => { + const result = server.call('provar.testcase.step.edit', { + test_case_path: tcPath, + mode: 'add', + test_item_id: '2', + position: 'before', + step_xml: NEW_STEP_XML, + validate_after_edit: false, + }); + + assert.equal(isError(result), false); + const written = fs.readFileSync(tcPath, 'utf-8'); + const pos99 = written.indexOf('testItemId="99"'); + const pos2 = written.indexOf('testItemId="2"'); + assert.ok(pos99 < pos2, 'new step should appear before anchor step 2'); + }); + + // STEP_NOT_FOUND — remove + it('mode=remove returns STEP_NOT_FOUND with all IDs when testItemId missing', () => { + const result = server.call('provar.testcase.step.edit', { + test_case_path: tcPath, + mode: 'remove', + test_item_id: '999', + }); + + assert.equal(isError(result), true); + const body = parseText(result); + assert.equal(body['error_code'], 'STEP_NOT_FOUND'); + const details = body['details'] as Record; + const allIds = details['all_test_item_ids'] as string[]; + assert.ok(Array.isArray(allIds), 'details.all_test_item_ids should be an array'); + assert.ok(allIds.includes('1'), 'should list testItemId 1'); + assert.ok(allIds.includes('2'), 'should list testItemId 2'); + assert.ok(allIds.includes('3'), 'should list testItemId 3'); + // File should be unchanged + const written = fs.readFileSync(tcPath, 'utf-8'); + assert.equal(written, VALID_TESTCASE_XML); + }); + + // STEP_NOT_FOUND — add + it('mode=add returns STEP_NOT_FOUND with all IDs when anchor missing', () => { + const result = server.call('provar.testcase.step.edit', { + test_case_path: tcPath, + mode: 'add', + test_item_id: '999', + step_xml: NEW_STEP_XML, + }); + + assert.equal(isError(result), true); + const body = parseText(result); + assert.equal(body['error_code'], 'STEP_NOT_FOUND'); + }); + + // INVALID_STEP_XML — step_xml contains no element + it('mode=add returns INVALID_STEP_XML when step_xml contains no element', () => { + const result = server.call('provar.testcase.step.edit', { + test_case_path: tcPath, + mode: 'add', + test_item_id: '1', + step_xml: '', + }); + + assert.equal(isError(result), true); + const body = parseText(result); + assert.equal(body['error_code'], 'INVALID_STEP_XML'); + // File must be untouched (no backup written) + assert.ok(!fs.existsSync(tcPath + '.bak'), 'no backup should be written for pre-mutation errors'); + const written = fs.readFileSync(tcPath, 'utf-8'); + assert.equal(written, VALID_TESTCASE_XML); + }); + + // INVALID_STEP_XML — step_xml contains multiple elements + it('mode=add returns INVALID_STEP_XML when step_xml contains multiple elements', () => { + const multiStep = + '' + + ''; + + const result = server.call('provar.testcase.step.edit', { + test_case_path: tcPath, + mode: 'add', + test_item_id: '1', + step_xml: multiStep, + }); + + assert.equal(isError(result), true); + const body = parseText(result); + assert.equal(body['error_code'], 'INVALID_STEP_XML'); + assert.ok(!fs.existsSync(tcPath + '.bak'), 'no backup should be written for pre-mutation errors'); + }); + + // mode=add with missing step_xml + it('mode=add returns MISSING_INPUT when step_xml is absent', () => { + const result = server.call('provar.testcase.step.edit', { + test_case_path: tcPath, + mode: 'add', + test_item_id: '1', + }); + + assert.equal(isError(result), true); + const body = parseText(result); + assert.equal(body['error_code'], 'MISSING_INPUT'); + }); + + // Backup restored when validation fails after mutation + it('restores backup and returns INVALID_XML_AFTER_EDIT when mutated file fails validation', () => { + // Write a test case where removing the only step will leave empty, + // but the XML itself is still structurally valid. We need a case where + // validate_after_edit=true fires and the result is invalid. + // The simplest trigger: create a test case where removing all 3 steps produces + // an empty element — validateTestCase passes this since it only checks presence. + // Instead, we test with an intentionally broken step_xml that results in an invalid + // testCase XML (step with non-UUID guid). + const brokenStepXml = + ''; + + const result = server.call('provar.testcase.step.edit', { + test_case_path: tcPath, + mode: 'add', + test_item_id: '1', + step_xml: brokenStepXml, + validate_after_edit: true, + }); + + assert.equal(isError(result), true); + const body = parseText(result); + assert.equal(body['error_code'], 'INVALID_XML_AFTER_EDIT'); + + // Original file should be restored + const written = fs.readFileSync(tcPath, 'utf-8'); + assert.ok(!written.includes('testItemId="99"'), 'broken step should not be in restored file'); + assert.ok(!fs.existsSync(tcPath + '.bak'), 'backup should be deleted after restore'); + }); + + // Path policy: path outside allowed paths is rejected + it('rejects test_case_path outside allowed paths', () => { + const restrictedServer = new MockMcpServer(); + registerAllTestCaseStepTools(restrictedServer as never, { allowedPaths: [path.join(tmpDir, 'allowed')] }); + fs.mkdirSync(path.join(tmpDir, 'allowed'), { recursive: true }); + + const result = restrictedServer.call('provar.testcase.step.edit', { + test_case_path: tcPath, // tcPath is in tmpDir root, not in allowed subdir + mode: 'remove', + test_item_id: '1', + }); + + assert.equal(isError(result), true); + const body = parseText(result); + assert.ok( + body['error_code'] === 'PATH_NOT_ALLOWED' || body['error_code'] === 'PATH_TRAVERSAL', + `expected path policy error, got: ${String(body['error_code'])}` + ); + }); + + // validate_after_edit=true with valid result includes validation in response + it('returns validation result in response when validate_after_edit=true and edit is valid', () => { + const result = server.call('provar.testcase.step.edit', { + test_case_path: tcPath, + mode: 'remove', + test_item_id: '2', + validate_after_edit: true, + }); + + assert.equal(isError(result), false); + const body = parseText(result); + assert.equal(body['success'], true); + // The fixture XML has a valid structure so validation should pass + const validation = body['validation'] as Record | undefined; + assert.ok(validation !== undefined, 'validation field should be present'); + }); + + // FILE_NOT_FOUND + it('returns FILE_NOT_FOUND when test case does not exist', () => { + const missing = path.join(tmpDir, 'nonexistent.testcase'); + const result = server.call('provar.testcase.step.edit', { + test_case_path: missing, + mode: 'remove', + test_item_id: '1', + }); + + assert.equal(isError(result), true); + const body = parseText(result); + assert.equal(body['error_code'], 'FILE_NOT_FOUND'); + }); +});