feat(validate): Wave 3 — XML generator correctness + 4 new local validator rules by mrdailey99 · Pull Request #136 · ProvarTesting/provardx-cli

mrdailey99 · 2026-05-01T19:13:12Z

Summary

Generator XML structure (testCaseGenerate.ts): target → class="uiTarget", locator → class="uiLocator", {Var} values → class="variable", SetValues → class="valueList"/<namedValues>. AssertValues now uses flat argument structure (not valueList — previously a false positive for ASSERT-001).
Four new local validator rules (testCaseValidate.ts): UI-TARGET-001, UI-LOCATOR-001, SETVALUES-STRUCTURE-001 (ERROR), VAR-REF-001 (WARNING). These mirror gaps confirmed in the quality-hub-agents backend rules.
Security: Added assertPathAllowed(planDir, config.allowedPaths) to testplan.create-plan path (D1 path traversal fix).
Docs: docs/mcp.md updated with argument convention table and all new rule IDs.
Version: 1.5.0-beta.11 → 1.5.0-beta.12

Test plan

node_modules/.bin/nyc node_modules/.bin/mocha "test/**/*.test.ts" — 829 passing, 1 pending
tsc --noEmit — clean
eslint src/mcp/tools/testCaseGenerate.ts src/mcp/tools/testCaseValidate.ts src/mcp/tools/testPlanTools.ts — clean
node scripts/mcp-smoke.cjs — all PASS (40 tools)
New validator rules fire correctly on malformed test case XML
AssertValues generate → flat args, no valueList structure
SetValues namedValues keys target/locator emit plain string (not uiTarget/uiLocator)

Backend gap note (quality-hub-agents)

A separate issue VAR-STRING-LITERAL-001 was identified in the quality-hub-agents repo: the backend ValidatorRegistry has no rule catching {VarName} stored as valueClass="string". An implementation overview has been prepared for the quality-hub-agents team.

🤖 Generated with Claude Code

Develop

docs(mcp): add npx alternative and fix Claude Code setup instructions

docs: update license terminology and improve error messages in docume…

Develop

PDX-0: (chore) package+server json sync

updated server.json for proper formatting per MCP requirements

added workflow section to publish to mcp registry

D1: add provar.testplan.create tool — creates plans/{name}/ directory and root .planitem; previously agents had to create plan files manually. D2: emit class="uiTarget" uri="..." for the 'target' argument and class="uiLocator" uri="..." for 'locator' argument in XML generation; previously both were written as valueClass="string" causing a ClassCastException in the Provar runtime. D3: SetValues / AssertValues steps now emit <value class="valueList" mutable="Mutable"><namedValues>...</namedValues></value> for the 'values' argument; previously a pipe-delimited string was written, causing an immediate ClassCastException. D4: attribute values matching {VarName} or {Obj.Field} are emitted as class="variable" with <path element="..."/> children; previously emitted as plain string literals which the runtime rejects. D7: provar.testcase.generate now includes a cleanup warning when ApexDeleteObject steps are present, explaining that cleanup steps after a failure point are skipped when stopOnError=false. Also updates StepSchema.attributes description and TOOL_DESCRIPTION to document all four special value conventions for agent grounding. Tests: 19 new assertions across testCaseGenerate.test.ts and testPlanTools.test.ts covering happy path, dry_run, and error cases. Smoke test: +1 entry for provar.testplan.create (TOTAL_EXPECTED 50->51). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…TRUCTURE-001, VAR-REF-001 rules Addresses Session 3 feedback D2 (uiTarget class), D3 (SetValues valueList structure), and D4 (variable reference detection). Mirrors quality-hub-agents backend rule IDs where they exist; VAR-REF-001 fills a gap present in both local and backend validation.

D1: add assertPathAllowed(planDir) in testPlanTools create-plan path D2: skip uiTarget/uiLocator dispatch inside SetValues namedValues context D3: remove AssertValues from SET_VALUES_APIS; update test to expect flat args Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… rules Add docs for UI-TARGET-001, UI-LOCATOR-001, SETVALUES-STRUCTURE-001, VAR-REF-001 Document generator XML argument conventions (uiTarget/uiLocator/variable/valueList) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Copilot

Pull request overview

This PR improves Provar MCP test case generation/validation correctness (XML argument typing + new local validation rules), adds a new provar.testplan.create tool with path policy enforcement, and updates docs/tests plus a beta version bump.

Changes:

Add provar.testplan.create tool, smoke coverage, and unit tests.
Update test case generator XML emission for uiTarget/uiLocator, variable refs, and SetValues/AssertValues argument structures; add validator rules to match.
Update docs and bump version to 1.5.0-beta.12.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
`src/mcp/tools/testPlanTools.ts`	Adds new `provar.testplan.create` tool and registers it in `registerAllTestPlanTools`.
`src/mcp/tools/testCaseValidate.ts`	Adds VAR-REF-001 plus new UI/SetValues structural rules and shared arg normalization helper.
`src/mcp/tools/testCaseGenerate.ts`	Updates argument XML generation for uiTarget/uiLocator, variable refs, SetValues structure, and adds cleanup warning.
`test/unit/mcp/testPlanTools.test.ts`	Adds unit tests for `provar.testplan.create` and updates registration coverage.
`test/unit/mcp/testCaseValidate.test.ts`	Adds tests for the 4 new local validator rules.
`test/unit/mcp/testCaseGenerate.test.ts`	Adds tests for uiTarget/uiLocator, SetValues/AssertValues structure, var refs, and cleanup warnings.
`docs/mcp.md`	Documents new argument conventions and new validator rule IDs.
`scripts/mcp-smoke.cjs`	Adds smoke invocation for `provar.testplan.create` and updates expected tool count.
`package.json`	Version bump to `1.5.0-beta.12`.
`server.json`	Version bump to `1.5.0-beta.12`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-05-01T19:18:18Z

+        '(2) SetValues / AssertValues: pass each variable name and its value as a flat key/value pair; ' +
+        '    the generator wraps them in <value class="valueList"><namedValues>...</namedValues></value> automatically. ' +
+        '    Example for SetValues: { "testCaseName": "TC_New", "testType": "Acceptance testing" } ' +
+        '(3) target argument (UiWithScreen / UiWithRow): pass the sf:ui:target or ui:pageobject:target URI; ' +
+        '    emitted as class="uiTarget" uri="...". ' +
+        '(4) locator argument (UiDoAction / UiAssert): pass the locator URI; emitted as class="uiLocator" uri="...". ' +


The StepSchema attributes description says SetValues / AssertValues are wrapped into valueList/namedValues automatically, but the generator now intentionally emits AssertValues using flat arguments (per tests and docs/mcp.md). Update this description to avoid instructing callers to pass AssertValues data in a format the generator no longer produces.

Suggested change

'(2) SetValues / AssertValues: pass each variable name and its value as a flat key/value pair; ' +

' the generator wraps them in <value class="valueList"><namedValues>...</namedValues></value> automatically. ' +

' Example for SetValues: { "testCaseName": "TC_New", "testType": "Acceptance testing" } ' +

'(3) target argument (UiWithScreen / UiWithRow): pass the sf:ui:target or ui:pageobject:target URI; ' +

' emitted as class="uiTarget" uri="...". ' +

'(4) locator argument (UiDoAction / UiAssert): pass the locator URI; emitted as class="uiLocator" uri="...". ' +

'(2) SetValues: pass each variable name and its value as a flat key/value pair; ' +

' the generator wraps them in <value class="valueList"><namedValues>...</namedValues></value> automatically. ' +

' Example for SetValues: { "testCaseName": "TC_New", "testType": "Acceptance testing" } ' +

'(3) AssertValues: pass assertions as flat key/value argument pairs; these are emitted as flat arguments, not wrapped into valueList/namedValues. ' +

'(4) target argument (UiWithScreen / UiWithRow): pass the sf:ui:target or ui:pageobject:target URI; ' +

' emitted as class="uiTarget" uri="...". ' +

'(5) locator argument (UiDoAction / UiAssert): pass the locator URI; emitted as class="uiLocator" uri="...". ' +

Copilot · 2026-05-01T19:18:18Z

+  'SetValues / AssertValues: pass named variable values as flat key/value pairs in attributes; ' +
+    'the generator wraps them in <value class="valueList"><namedValues>...</namedValues></value> automatically.',


TOOL_DESCRIPTION still states that "SetValues / AssertValues" are wrapped into valueList/namedValues automatically, but AssertValues generation is now flat (and namedValues is treated as a validation warning via ASSERT-001). This text should be updated to reflect the new AssertValues behavior to prevent consumers from relying on the wrong structure.

Suggested change

'SetValues / AssertValues: pass named variable values as flat key/value pairs in attributes; ' +

'the generator wraps them in <value class="valueList"><namedValues>...</namedValues></value> automatically.',

'SetValues: pass named variable values as flat key/value pairs in attributes; ' +

'the generator wraps them in <value class="valueList"><namedValues>...</namedValues></value> automatically.',

'AssertValues: pass assertion values as flat key/value pairs in attributes; they are generated flat and not wrapped in namedValues.',

'If AssertValues uses namedValues-shaped content, validation reports warning ASSERT-001.',

Copilot · 2026-05-01T19:18:19Z


+// APIs whose 'values' argument must use class="valueList"/<namedValues> (D3).
+const SET_VALUES_APIS = new Set([
+  SHORTHAND_TO_FQID['SetValues'] ?? '', // com.provar.plugins.bundled.apis.control.SetValues


SET_VALUES_APIS only contains the bundled SetValues API ID. Because the tool schema explicitly allows callers to pass fully-qualified IDs directly, passing another valid SetValues ID (e.g. com.provar.plugins.core.data.SetValues, which the validator/tests recognize) would bypass buildSetValuesXml and generate an invalid flat structure. Consider broadening the check (e.g. apiId.endsWith('.SetValues') && !apiId.endsWith('.AssertValues')) or including the additional known FQIDs in SET_VALUES_APIS.

Suggested change

SHORTHAND_TO_FQID['SetValues'] ?? '', // com.provar.plugins.bundled.apis.control.SetValues

SHORTHAND_TO_FQID['SetValues'] ?? '', // com.provar.plugins.bundled.apis.control.SetValues

'com.provar.plugins.core.data.SetValues',

Copilot · 2026-05-01T19:18:19Z

+  if (!inNamedValues) {
+    // D2: 'target' argument (UiWithScreen / UiWithRow) → class="uiTarget".
+    if (key === 'target') {
+      return `${indent}<value class="uiTarget" uri="${escapeXmlAttr(val)}"/>`;
+    }
+    // D2: 'locator' argument (UiDoAction / UiAssert) → class="uiLocator".
+    if (key === 'locator') {
+      return `${indent}<value class="uiLocator" uri="${escapeXmlAttr(val)}"/>`;
+    }


buildArgumentValue applies the uiTarget/uiLocator dispatch purely based on the argument key name. That doesn't match the surrounding docs (which scope these conversions to specific UI APIs), and it means any non-UI step with an argument named "target" or "locator" would get a UI-specific XML class. Consider making this conversion conditional on the resolved apiId (e.g. pass apiId context into buildArgumentsXml/buildArgumentValue) or updating the docs to reflect the actual behavior.

Copilot · 2026-05-01T19:18:19Z

 - **DATA-001** — `testCase` declares a `<dataTable>` element. CLI standalone execution does not bind CSV column variables; steps using variable references will resolve to null. Use `SetValues` (Test scope) steps instead, or add the test to a test plan.
 - **ASSERT-001** — An `AssertValues` step uses the `argument id="values"` (namedValues) format, which is designed for UI element attribute assertions. For Apex/SOQL result or variable comparisons this silently passes as `null=null`. Use separate `expectedValue`, `actualValue`, and `comparisonType` arguments instead.
+- **UI-TARGET-001** — A UiWithScreen or UiWithRow `target` argument uses the wrong XML class (e.g. `class="value"`). Must be `class="uiTarget"` or the screen binding is silently ignored at runtime.
+- **UI-LOCATOR-001** — A UiDoAction or UiAssert `locator` argument uses the wrong XML class. Must be `class="uiLocator"` or Provar cannot resolve the element.


UI-LOCATOR-001 in code also applies to UiScrollToElement, but this doc entry only mentions UiDoAction/UiAssert. Please update the docs to include UiScrollToElement (and keep it consistent with the validator implementation).

Suggested change

- **UI-LOCATOR-001** — A UiDoAction or UiAssert `locator` argument uses the wrong XML class. Must be `class="uiLocator"` or Provar cannot resolve the element.

- **UI-LOCATOR-001** — A UiDoAction, UiAssert, or UiScrollToElement `locator` argument uses the wrong XML class. Must be `class="uiLocator"` or Provar cannot resolve the element.

Copilot · 2026-05-01T19:18:19Z

+| Argument key / value pattern         | Emitted XML class             | API context                         |
+| ------------------------------------ | ----------------------------- | ----------------------------------- |
+| `target` key                         | `class="uiTarget"`            | UiWithScreen, UiWithRow             |
+| `locator` key                        | `class="uiLocator"`           | UiDoAction, UiAssert                |
+| Value matches `{VarName}` or `{A.B}` | `class="variable"` + `<path>` | Any step                            |
+| SetValues attributes                 | `class="valueList"/<namedValues>` | SetValues only                  |
+| All other values                     | `class="value" valueClass="string"` | Any step                      |


The argument convention table lists locator conversion as applying to UiDoAction/UiAssert only, but the validator rule UI-LOCATOR-001 also checks UiScrollToElement. To avoid confusion, consider adding UiScrollToElement to this table row (or align the validator scope).

Suggested change

| Argument key / value pattern | Emitted XML class | API context |

| ------------------------------------ | ----------------------------- | ----------------------------------- |

| `target` key | `class="uiTarget"` | UiWithScreen, UiWithRow |

| `locator` key | `class="uiLocator"` | UiDoAction, UiAssert |

| Value matches `{VarName}` or `{A.B}` | `class="variable"` + `<path>` | Any step |

| SetValues attributes | `class="valueList"/<namedValues>` | SetValues only |

| All other values | `class="value" valueClass="string"` | Any step |

| Argument key / value pattern | Emitted XML class | API context |

| ------------------------------------ | ----------------------------- | ---------------------------------------------- |

| `target` key | `class="uiTarget"` | UiWithScreen, UiWithRow |

| `locator` key | `class="uiLocator"` | UiDoAction, UiAssert, UiScrollToElement |

| Value matches `{VarName}` or `{A.B}` | `class="variable"` + `<path>` | Any step |

| SetValues attributes | `class="valueList"/<namedValues>` | SetValues only |

| All other values | `class="value" valueClass="string"` | Any step |

Copilot · 2026-05-01T19:18:20Z

+        const planDir = path.join(projectRoot, 'plans', plan_name);
+        assertPathAllowed(planDir, config.allowedPaths);
+        const planItemPath = path.join(planDir, '.planitem');


plan_name is used in path.join(projectRoot, 'plans', plan_name). If plan_name is an absolute path (or contains path separators), path.join will ignore the projectRoot/plans prefix and allow writing the .planitem outside the project directory (still potentially within allowedPaths). Consider validating plan_name as a single directory name (no '/' or '\', not absolute) and/or enforcing that the resolved planDir is within path.join(projectRoot,'plans').

- Separate SetValues/AssertValues in StepSchema and TOOL_DESCRIPTION - Replace SET_VALUES_APIS Set with resolvedApiId.includes('SetValues') (mirrors validator) - Make target/locator dispatch API-aware (pass resolvedApiId through buildArgumentsXml) - Fix buildUiWithScreenXml to pass wrapperApiId so target emits uiTarget class - Validate plan_name has no path separators to prevent path.join injection - Add UiScrollToElement to UI-LOCATOR-001 docs and argument convention table Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…vidence) UiScrollToElement has zero usage in the internal test corpus and no documented locator argument. The inclusion was speculative — reverting to UiDoAction/UiAssert only. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

mrdailey99 and others added 14 commits April 7, 2026 12:33

Merge pull request #107 from ProvarTesting/develop

37213d9

Develop

Merge pull request #117 from ProvarTesting/develop

c62c6ca

Develop

Merge pull request #119 from ProvarTesting/develop

1ef9f6a

Develop

Merge pull request #124 from ProvarTesting/develop

a3639d8

docs(mcp): add npx alternative and fix Claude Code setup instructions

Merge pull request #125 from ProvarTesting/develop

184a98a

docs: update license terminology and improve error messages in docume…

Merge pull request #127 from ProvarTesting/develop

7ba1616

Develop

Merge pull request #129 from ProvarTesting/develop

ef06d73

PDX-0: (chore) package+server json sync

Merge pull request #130 from ProvarTesting/develop

63acc2c

updated server.json for proper formatting per MCP requirements

Merge pull request #131 from ProvarTesting/develop

e7d557b

added workflow section to publish to mcp registry

PDX-0: chore: fix duplicate comment numbers in mcp-smoke.cjs

7f7ea1c

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings May 1, 2026 19:13

Copilot started reviewing on behalf of mrdailey99 May 1, 2026 19:13 View session

Copilot AI reviewed May 1, 2026

View reviewed changes

mrdailey99 and others added 2 commits May 1, 2026 14:57

mrdailey99 merged commit 416a6cc into develop May 1, 2026
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(validate): Wave 3 — XML generator correctness + 4 new local validator rules#136

feat(validate): Wave 3 — XML generator correctness + 4 new local validator rules#136
mrdailey99 merged 16 commits intodevelopfrom
feature/wave3-editing-tools

mrdailey99 commented May 1, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI May 1, 2026

Uh oh!

Copilot AI May 1, 2026

Uh oh!

Copilot AI May 1, 2026

Uh oh!

Copilot AI May 1, 2026

Uh oh!

Copilot AI May 1, 2026

Uh oh!

Copilot AI May 1, 2026

Uh oh!

Copilot AI May 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		'SetValues / AssertValues: pass named variable values as flat key/value pairs in attributes; ' +
		'the generator wraps them in <value class="valueList"><namedValues>...</namedValues></value> automatically.',

	SHORTHAND_TO_FQID['SetValues'] ?? '', // com.provar.plugins.bundled.apis.control.SetValues
	SHORTHAND_TO_FQID['SetValues'] ?? '', // com.provar.plugins.bundled.apis.control.SetValues
	'com.provar.plugins.core.data.SetValues',

	- UI-LOCATOR-001 — A UiDoAction or UiAssert `locator` argument uses the wrong XML class. Must be `class="uiLocator"` or Provar cannot resolve the element.
	- UI-LOCATOR-001 — A UiDoAction, UiAssert, or UiScrollToElement `locator` argument uses the wrong XML class. Must be `class="uiLocator"` or Provar cannot resolve the element.

Conversation

mrdailey99 commented May 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Backend gap note (quality-hub-agents)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI May 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mrdailey99 commented May 1, 2026 •

edited

Loading