diff --git a/CHANGELOG.md b/CHANGELOG.md index 942d46c..6042668 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,25 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] +## [0.9.6] - 2026-05-12 + +### Fixed +- **`flutter_probe_gen`: `Mock` path silently truncated.** The emitter wrote the path unquoted (`when the app calls GET /api/products`), so the Go lexer split on `/` and the parser only recorded the first IDENT segment. Now emits the canonical quoted form. Caught by a new `mock_and_call` golden + the existing cross-language integration test. +- **`flutter_probe_gen`: `See` suffixes silently dropped.** When `state`, `containing`, and `matching` were all set on a single assertion, only the last branch's text reached the output. Now composes all three suffixes additively: `see "x" is enabled contains "y" matching "z"`. Caught by a new `see_states` golden covering the matrix. + +### Added +- **`flutter_probe_annotation`: `@ProbeCompositeTest` annotation.** The flagship multi-device composite testing feature finally has a DSL surface. Pair with `Device(alias, target: …)`, `OnDevice(alias, steps: […])` per-device groups, and `Sync(label)` cross-device barriers. Emitter generates standard `composite test` / `devices` / `:` / `sync` blocks that the existing CLI runner picks up unchanged. +- **`flutter_probe_annotation`: `See.id` / `See.selector` factories** — assertions can now target by `ValueKey` or any rich selector (Ordinal, Below/Above/LeftOf/RightOf, InContainer, TypeSel) rather than only by literal visible text. Same factories on `DontSee`. The Go parser always supported this; the DSL just didn't expose it. +- **`flutter_probe_annotation`: `WaitUntil.idAppears` / `.idDisappears`** — emits unquoted `#key` selector form (Go parser's WaitSelector branch), which is more reliable than text matching for stable `ValueKey`-tagged widgets. +- **`flutter_probe_gen`: 6 new golden fixtures.** `mock_and_call`, `see_states`, `composite_chat`, `wait_variants`, `examples_inline`, `kitchen_sink`. The kitchen sink fixture exercises one of every step, selector kind, and control-flow construct. Every fixture round-trips through `internal/parser/golden_integration_test.go`. Total golden coverage went from 4 → 10 fixtures, builder tests from 5 → 11. + +### Changed +- **`flutter_probe_annotation`: `Press` and `Pinch` are now `@Deprecated`.** The Go parser has no `press` or `pinch` case, so emitted text fell through to `parseRecipeCall` and was misinterpreted. Marked deprecated until runtime support lands. Use `GoBack()` in place of `Press('back')`. +- **`flutter_probe_gen`: emitter no longer coupled to enum declaration order.** `_direction`, `_httpMethod`, and the `See` state lookup now read the enum constant identifier (`_name` field) instead of indexing a hard-coded array by `.index`. Reordering `Direction`, `HttpMethod`, or `SeeState` no longer silently corrupts emitted ProbeScript. + +### Docs +- New website page: [Annotation-driven Tests](https://flutterprobe.dev/probescript/annotations/) — full reference for the annotation DSL with every step class, selector kind, and the new composite test syntax. + ## [0.9.5] - 2026-05-12 ### Fixed diff --git a/README.md b/README.md index dec2d5b..f35ca8f 100644 --- a/README.md +++ b/README.md @@ -369,18 +369,18 @@ run dart: Co-locate `.probe` tests with the Flutter widgets they exercise. Two Dart packages handle this: -- **`flutter_probe_annotation`** — `@ProbeSuite`, `@ProbeTest`, `@ProbeRecipe` decorators plus a fully type-checked step DSL (all 31 ProbeScript verbs, all 6 selector kinds, hooks, loops, conditionals, recipes, examples). +- **`flutter_probe_annotation`** — `@ProbeSuite`, `@ProbeTest`, `@ProbeRecipe`, `@ProbeCompositeTest` decorators plus a fully type-checked step DSL (all 31 ProbeScript verbs, all 6 selector kinds, hooks, loops, conditionals, recipes, examples, multi-device composite tests). - **`flutter_probe_gen`** — a `build_runner` builder that reads the annotations and emits matching `.probe` files into `tests/generated/`. Add to your Flutter app's `pubspec.yaml`: ```yaml dependencies: - flutter_probe_annotation: ^0.9.3 - flutter_probe_agent: ^0.9.3 + flutter_probe_annotation: ^0.9.6 + flutter_probe_agent: ^0.9.6 dev_dependencies: - flutter_probe_gen: ^0.9.3 + flutter_probe_gen: ^0.9.6 build_runner: ^2.15.0 ``` @@ -415,7 +415,9 @@ probe test tests/ # picks up tests/generated/login_screen.probe Test definitions are now type-checked by `flutter analyze` — a misspelt step name is a compile error rather than a runtime surprise. Selectors stay in sync with widget code because they live in the same file. The generated `.probe` file goes through the same parser, agent, and reporter as a hand-written one. -Full reference: [`docs/wiki/Annotations.md`](docs/wiki/Annotations.md). +**v0.9.6** completes the annotation surface: full composite-test DSL (`@ProbeCompositeTest`, `Device`, `OnDevice`, `Sync`), id/selector-based `See`/`DontSee`, `WaitUntil.idAppears`, and composable `state` + `containing` + `matching` assertions. Plus fixes for two emitter bugs (`Mock` paths and `See` suffix dropping). + +Full reference: [flutterprobe.dev/probescript/annotations](https://flutterprobe.dev/probescript/annotations/) (or [`docs/wiki/Annotations.md`](docs/wiki/Annotations.md) on GitHub). ## CLI Commands diff --git a/docs/wiki/Annotations.md b/docs/wiki/Annotations.md index f43571c..8fb6ff1 100644 --- a/docs/wiki/Annotations.md +++ b/docs/wiki/Annotations.md @@ -28,11 +28,11 @@ tests. Annotations: ```yaml # pubspec.yaml dependencies: - flutter_probe_annotation: ^0.9.3 - flutter_probe_agent: ^0.9.3 + flutter_probe_annotation: ^0.9.6 + flutter_probe_agent: ^0.9.6 dev_dependencies: - flutter_probe_gen: ^0.9.3 + flutter_probe_gen: ^0.9.6 build_runner: ^2.15.0 ``` diff --git a/docs/wiki/Home.md b/docs/wiki/Home.md index 9aee097..7785d44 100644 --- a/docs/wiki/Home.md +++ b/docs/wiki/Home.md @@ -18,7 +18,7 @@ Welcome to the FlutterProbe wiki. This documentation covers architecture details ## Project Status -FlutterProbe is in active development. Current version: **0.9.5**. +FlutterProbe is in active development. Current version: **0.9.6**. ### Repository Structure diff --git a/internal/parser/golden_integration_test.go b/internal/parser/golden_integration_test.go index 682c70f..35a40c7 100644 --- a/internal/parser/golden_integration_test.go +++ b/internal/parser/golden_integration_test.go @@ -49,9 +49,11 @@ func TestGoldenIntegration_DartEmittedFiles(t *testing.T) { t.Fatalf("parser rejected golden %s:\n%v\n--- contents ---\n%s", filepath.Base(path), err, string(data)) } - // Sanity: every golden defines at least one test, recipe, or hook. - if len(prog.Tests) == 0 && len(prog.Recipes) == 0 && len(prog.Hooks) == 0 { - t.Errorf("golden %s parsed but produced no tests/recipes/hooks", + // Sanity: every golden defines at least one test, composite test, + // recipe, or hook. + if len(prog.Tests) == 0 && len(prog.CompositeTests) == 0 && + len(prog.Recipes) == 0 && len(prog.Hooks) == 0 { + t.Errorf("golden %s parsed but produced no tests/composite tests/recipes/hooks", filepath.Base(path)) } }) diff --git a/probe_agent/CHANGELOG.md b/probe_agent/CHANGELOG.md index 4b8819e..8b4937a 100644 --- a/probe_agent/CHANGELOG.md +++ b/probe_agent/CHANGELOG.md @@ -1,5 +1,11 @@ # Changelog +## 0.9.6 - 2026-05-12 + +- Version bump to match CLI v0.9.6. No agent code changes — annotation DSL + completeness work is in the flutter_probe_annotation & + flutter_probe_gen packages. + ## 0.9.5 - 2026-05-12 - **Fix: iOS/Impeller screenshots** — `take_screenshot` previously called diff --git a/probe_agent/pubspec.yaml b/probe_agent/pubspec.yaml index 47fcea3..71d1572 100644 --- a/probe_agent/pubspec.yaml +++ b/probe_agent/pubspec.yaml @@ -3,7 +3,7 @@ description: >- On-device E2E test agent for FlutterProbe. Embeds in your Flutter app and executes test commands via direct widget-tree access with sub-50ms latency. -version: 0.9.5 +version: 0.9.6 homepage: https://flutterprobe.dev repository: https://github.com/AlphaWaveSystems/flutter-probe issue_tracker: https://github.com/AlphaWaveSystems/flutter-probe/issues diff --git a/probe_annotation/CHANGELOG.md b/probe_annotation/CHANGELOG.md index 54d7ca9..13e5628 100644 --- a/probe_annotation/CHANGELOG.md +++ b/probe_annotation/CHANGELOG.md @@ -1,5 +1,36 @@ # Changelog +## 0.9.6 - 2026-05-12 + +### Added + +- **`@ProbeCompositeTest`** — declare multi-device composite tests as + annotations. Pair with `Device(alias, target: ...)` declarations, + `OnDevice(alias, steps: [...])` per-device step groups, and + `Sync(label)` cross-device barriers. The generated `.probe` block + uses the standard `composite test` / `devices` / `sync` syntax that + the CLI runner already understands. +- **`See.id(key)` / `See.selector(Selector)`** — target assertions by + `ValueKey` or by rich selector (Ordinal, Below/Above/LeftOf/RightOf, + InContainer, TypeSel). Same factories on `DontSee`. Previously, + `See`/`DontSee` only accepted a literal text string — the parser + always supported every selector kind but the DSL didn't expose it. +- **`WaitUntil.idAppears(key)` / `.idDisappears(key)`** — emit + unquoted `wait until #key appears`, exercising the Go parser's + WaitSelector branch. More reliable than text matching for widgets + with stable `ValueKey`s. +- **Composable `See` suffixes** — `See('x', state: SeeState.enabled, + containing: 'y')` now emits `see "x" is enabled contains "y"` + (both suffixes present). Previously the second suffix silently + overwrote the first. + +### Changed + +- **`Press` and `Pinch` are now `@Deprecated`** with a clear note — + the Go-side parser has no `press` or `pinch` case and emitted text + would be misparsed as a recipe call. They'll be re-enabled when + runtime support lands. Use `GoBack()` in place of `Press('back')`. + ## 0.9.5 - 2026-05-12 - Version bump to match CLI v0.9.5. No annotation API changes. diff --git a/probe_annotation/lib/src/annotations.dart b/probe_annotation/lib/src/annotations.dart index 0f1e1f3..a88289a 100644 --- a/probe_annotation/lib/src/annotations.dart +++ b/probe_annotation/lib/src/annotations.dart @@ -114,3 +114,70 @@ class ProbeRecipe { this.steps = const [], }); } + +/// Declares a multi-device composite test. The annotated class becomes a +/// `composite test "name"` block in the generated `.probe` file. +/// +/// Devices are declared by alias (`A`, `B`, `Sender`, etc.) and steps are +/// scoped per-device using [OnDevice]. [Sync] barriers between [OnDevice] +/// groups force every device to reach the same checkpoint before any +/// device proceeds. +/// +/// Example — chat between two users on two simulators: +/// +/// ```dart +/// @ProbeCompositeTest( +/// name: 'alice sends bob a message', +/// tags: ['composite', 'smoke'], +/// devices: [ +/// Device('A', target: 'iPhone 15 Simulator'), +/// Device('B', target: 'Pixel 9 Emulator'), +/// ], +/// body: [ +/// OnDevice('A', steps: [ +/// Open(), +/// Tap(text: 'Sign in as Alice'), +/// ]), +/// OnDevice('B', steps: [ +/// Open(), +/// Tap(text: 'Sign in as Bob'), +/// ]), +/// Sync('both signed in'), +/// OnDevice('A', steps: [ +/// Tap(text: 'New message'), +/// Type('hello bob'), +/// Tap(text: 'Send'), +/// ]), +/// OnDevice('B', steps: [ +/// WaitUntil.appears('hello bob'), +/// See('hello bob'), +/// ]), +/// ], +/// ) +/// class ChatComposite {} +/// ``` +class ProbeCompositeTest { + final String name; + final List tags; + final List devices; + + /// Composite body — must contain only [OnDevice] and [Sync] elements. + final List body; + + const ProbeCompositeTest({ + required this.name, + this.tags = const [], + this.devices = const [], + this.body = const [], + }); +} + +/// One device entry in a [ProbeCompositeTest.devices] list. The [alias] +/// is referenced by [OnDevice]; [target] is an optional human-readable +/// device name shown in the generated `.probe` header and in failure +/// messages. +class Device { + final String alias; + final String? target; + const Device(this.alias, {this.target}); +} diff --git a/probe_annotation/lib/src/steps.dart b/probe_annotation/lib/src/steps.dart index 62ff4db..3924eca 100644 --- a/probe_annotation/lib/src/steps.dart +++ b/probe_annotation/lib/src/steps.dart @@ -75,6 +75,11 @@ class LongPress extends Step { } /// Emits: `press home` / `press back` / `press `. +/// +/// Not yet supported by the FlutterProbe runtime — the Go-side parser has +/// no `press` case and the emitted text will be misparsed as a recipe +/// call. Will be re-enabled in a future release that wires runtime support. +@Deprecated('Not yet supported by the runtime — coming in a future release') class Press extends Step { final String key; // home | back | volume_up | volume_down | … const Press(this.key); @@ -139,6 +144,11 @@ class Drag extends Step { } /// Emits: `pinch in` / `pinch out`. +/// +/// Not yet supported by the FlutterProbe runtime — the Go-side parser has +/// no `pinch` case and the emitted text will be misparsed as a recipe +/// call. Will be re-enabled in a future release that wires runtime support. +@Deprecated('Not yet supported by the runtime — coming in a future release') class Pinch extends Step { final bool zoomIn; const Pinch({this.zoomIn = false}); @@ -261,27 +271,99 @@ class Store extends Step { /// State checks for [See] assertions. enum SeeState { none, enabled, disabled, checked, focused } -/// Emits: `see "X"`, `see exactly N "X"`, `see "X" enabled`, -/// `see "X" containing "Y"`, `see "X" matching "regex"`. +/// Emits ProbeScript assertions of the form: +/// +/// see "X" +/// see exactly N "X" +/// see "X" is enabled +/// see "X" contains "Y" +/// see "X" matching "regex" +/// see #id is focused +/// +/// `state`, `containing`, and `matching` are all suffixes that can coexist +/// — `See('Welcome', state: SeeState.enabled, containing: 'world')` emits +/// `see "Welcome" is enabled contains "world"`. Use the [See.id] / +/// [See.selector] factories to target by `ValueKey` or arbitrary selector +/// instead of visible text. class See extends Step { - final String text; + final String? text; + final String? id; + final Selector? selector; final SeeState state; final String? containing; final String? matching; final int? exactly; const See( - this.text, { + String this.text, { this.state = SeeState.none, this.containing, this.matching, this.exactly, - }); + }) : id = null, + selector = null; + + /// Target a widget by its `ValueKey` or `Semantics.identifier`. + const See.id( + String this.id, { + this.state = SeeState.none, + this.containing, + this.matching, + this.exactly, + }) : text = null, + selector = null; + + /// Target a widget via an arbitrary [Selector] — ordinal, positional, + /// relational, or by widget type. + const See.selector( + Selector this.selector, { + this.state = SeeState.none, + this.containing, + this.matching, + this.exactly, + }) : text = null, + id = null; } -/// Emits: `don't see "X"`. +/// Emits negative assertions: +/// +/// don't see "X" +/// don't see #id +/// +/// Like [See], supports targeting by text, id, or arbitrary [Selector]. class DontSee extends Step { - final String text; - const DontSee(this.text); + final String? text; + final String? id; + final Selector? selector; + final SeeState state; + final String? containing; + final String? matching; + final int? exactly; + const DontSee( + String this.text, { + this.state = SeeState.none, + this.containing, + this.matching, + this.exactly, + }) : id = null, + selector = null; + + const DontSee.id( + String this.id, { + this.state = SeeState.none, + this.containing, + this.matching, + this.exactly, + }) : text = null, + selector = null; + + const DontSee.selector( + Selector this.selector, { + this.state = SeeState.none, + this.containing, + this.matching, + this.exactly, + }) : text = null, + id = null; } // ---- Wait variants ---- @@ -310,12 +392,25 @@ class WaitForAnimations extends Wait { const WaitForAnimations(); } +/// Wait until a target widget appears or disappears. +/// +/// WaitUntil.appears('Dashboard') → wait until "Dashboard" appears +/// WaitUntil.disappears('Loading') → wait until "Loading" disappears +/// WaitUntil.idAppears('login_form') → wait until #login_form appears +/// WaitUntil.idDisappears('spinner') → wait until #spinner disappears +/// +/// The id-based factories emit unquoted `#key` selectors and exercise the +/// Go parser's WaitSelector branch — more reliable than text matching +/// for ValueKey-tagged widgets. class WaitUntil extends Wait { final String target; - final bool appears; // false → disappears - const WaitUntil._(this.target, this.appears); - const WaitUntil.appears(String target) : this._(target, true); - const WaitUntil.disappears(String target) : this._(target, false); + final bool appears; + final bool byId; + const WaitUntil._(this.target, this.appears, this.byId); + const WaitUntil.appears(String target) : this._(target, true, false); + const WaitUntil.disappears(String target) : this._(target, false, false); + const WaitUntil.idAppears(String key) : this._(key, true, true); + const WaitUntil.idDisappears(String key) : this._(key, false, true); } // ---- Control flow (block steps) ---- @@ -378,3 +473,27 @@ class RecipeStep extends Step { final List args; const RecipeStep(this.name, {this.args = const []}); } + +// ---- Composite test steps (only valid inside ProbeCompositeTest.body) ---- + +/// Scopes a group of [steps] to a single device in a composite test. The +/// [alias] must match a [Device.alias] declared in the enclosing +/// `@ProbeCompositeTest.devices`. +/// +/// Emits an `:` header followed by the indented step body. Multiple +/// `OnDevice` entries for the same alias accumulate, separated by [Sync] +/// barriers. +class OnDevice extends Step { + final String alias; + final List steps; + const OnDevice(this.alias, {this.steps = const []}); +} + +/// Cross-device barrier in a composite test. All devices must reach the +/// same [Sync] label before any device proceeds past it. +/// +/// Emits `sync "label"` at the composite body level. +class Sync extends Step { + final String label; + const Sync(this.label); +} diff --git a/probe_annotation/pubspec.yaml b/probe_annotation/pubspec.yaml index 50ad651..5809e82 100644 --- a/probe_annotation/pubspec.yaml +++ b/probe_annotation/pubspec.yaml @@ -3,7 +3,7 @@ description: >- Dart annotations for FlutterProbe — declare ProbeScript end-to-end tests as decorators on your Flutter screen classes. Pair with flutter_probe_gen to generate .probe test files at build time. -version: 0.9.5 +version: 0.9.6 homepage: https://flutterprobe.dev repository: https://github.com/AlphaWaveSystems/flutter-probe issue_tracker: https://github.com/AlphaWaveSystems/flutter-probe/issues diff --git a/probe_gen/CHANGELOG.md b/probe_gen/CHANGELOG.md index 48ea871..f693970 100644 --- a/probe_gen/CHANGELOG.md +++ b/probe_gen/CHANGELOG.md @@ -1,5 +1,48 @@ # Changelog +## 0.9.6 - 2026-05-12 + +### Fixed + +- **`Mock` path is now quoted** in the emitted `when the app calls ...` + line. Previously the path was unquoted, so the Go lexer split on `/` + and the parser recorded only the first IDENT segment — `Mock(path: + '/api/products')` silently became `/api`. Now emits `when the app + calls GET "/api/products"` which round-trips correctly. +- **`See` state/containing/matching suffixes compose additively.** + `See('x', state: SeeState.enabled, containing: 'y')` previously + emitted only `see "x" contains "y"` (state silently dropped). Now + emits `see "x" is enabled contains "y"` — all suffixes coexist as + the parser supports. + +### Added + +- **`@ProbeCompositeTest` emission** — new `emitCompositeTest` walks + `devices`, `OnDevice` groups, and `Sync` barriers, producing the + standard `composite test` / `devices` / `:` / `sync` block + layout. +- **`See.id` / `See.selector` rendering** — the emitter now reads the + optional `id` and `selector` fields on `See`/`DontSee` and renders + the appropriate target form. Text remains the default when no id + or selector is provided. +- **`WaitUntil.idAppears` / `.idDisappears` rendering** — emits + unquoted `#key` (selector form) when the DSL's `byId` flag is set. +- **6 new golden fixtures** in `probe_gen/test/fixtures/`: + `mock_and_call`, `see_states`, `composite_chat`, `wait_variants`, + `examples_inline`, `kitchen_sink`. The kitchen sink fixture + exercises one of every step, selector kind, and control-flow + construct. Every fixture round-trips through the Go-side parser + via `internal/parser/golden_integration_test.go`. + +### Changed + +- **Emitter is no longer coupled to enum declaration order.** + `_direction`, `_httpMethod`, and the `See` state name lookup now + read the enum constant identifier (`_name` field) rather than + indexing into a hard-coded array by `.index`. Reordering or + inserting values in `Direction`, `HttpMethod`, or `SeeState` no + longer silently corrupts emitted ProbeScript. + ## 0.9.5 - 2026-05-12 - Version bump to match CLI v0.9.5. No annotation API changes. diff --git a/probe_gen/lib/src/probe_builder.dart b/probe_gen/lib/src/probe_builder.dart index 5d71c16..9d1b6b5 100644 --- a/probe_gen/lib/src/probe_builder.dart +++ b/probe_gen/lib/src/probe_builder.dart @@ -1,5 +1,6 @@ /// The build_runner [Builder] that scans annotated Dart libraries for -/// `@ProbeSuite` / `@ProbeTest` / `@ProbeRecipe` and emits matching `.probe` +/// `@ProbeSuite` / `@ProbeTest` / `@ProbeRecipe` / `@ProbeCompositeTest` +/// and emits matching `.probe` /// files into `tests/generated/`. library; @@ -26,7 +27,8 @@ class ProbeBuilder implements Builder { final source = await buildStep.readAsString(buildStep.inputId); if (!source.contains('@ProbeSuite') && !source.contains('@ProbeTest') && - !source.contains('@ProbeRecipe')) { + !source.contains('@ProbeRecipe') && + !source.contains('@ProbeCompositeTest')) { return; } @@ -57,6 +59,9 @@ class ProbeBuilder implements Builder { case 'ProbeRecipe': emitter.emitRecipe(value); break; + case 'ProbeCompositeTest': + emitter.emitCompositeTest(value); + break; } } } diff --git a/probe_gen/lib/src/probe_emitter.dart b/probe_gen/lib/src/probe_emitter.dart index 9cfddb0..365c3a0 100644 --- a/probe_gen/lib/src/probe_emitter.dart +++ b/probe_gen/lib/src/probe_emitter.dart @@ -111,6 +111,53 @@ class ProbeEmitter { _hasContent = true; } + /// Emits a `composite test "name"` block from a `@ProbeCompositeTest` + /// annotation. Devices declaration, then a body of `:` blocks + /// (rendered from [OnDevice] entries) and `sync "label"` lines + /// (rendered from [Sync] entries). + void emitCompositeTest(DartObject composite) { + final name = _str(composite, 'name') ?? 'unnamed'; + final tags = _strList(composite, 'tags'); + final devices = _objList(composite, 'devices'); + final body = _objList(composite, 'body'); + + _buf.writeln('composite test "${_escape(name)}"'); + if (tags.isNotEmpty) { + _buf.writeln(' ${tags.map((t) => '@$t').join(' ')}'); + } + if (devices.isNotEmpty) { + _buf.writeln(' devices'); + for (final d in devices) { + final alias = _str(d, 'alias') ?? ''; + final target = _str(d, 'target') ?? ''; + if (target.isEmpty) { + _buf.writeln(' $alias:'); + } else { + _buf.writeln(' $alias: $target'); + } + } + } + for (final element in body) { + final t = _typeNameOf(element); + if (t == 'OnDevice') { + final alias = _str(element, 'alias') ?? ''; + final steps = _objList(element, 'steps'); + if (alias.isEmpty || steps.isEmpty) continue; + _buf.writeln(' $alias:'); + for (final s in steps) { + _emitStep(s, 2); + } + } else if (t == 'Sync') { + final label = _str(element, 'label') ?? ''; + _buf.writeln(' sync "${_escape(label)}"'); + } + // Other Step types in a composite body are ignored — DSL doc says + // body must contain only OnDevice and Sync. + } + _buf.writeln(); + _hasContent = true; + } + // ---- Step dispatch ---- void _emitStep(DartObject step, int depth) { @@ -258,7 +305,7 @@ class ProbeEmitter { _emitSee(step, depth); break; case 'DontSee': - _buf.writeln('${indent}don\'t see "${_escape(_str(step, 'text') ?? '')}"'); + _emitSee(step, depth, negated: true); break; // Wait @@ -277,7 +324,12 @@ class ProbeEmitter { case 'WaitUntil': final target = _str(step, 'target') ?? ''; final appears = _bool(step, 'appears'); - _buf.writeln('${indent}wait until "${_escape(target)}" ' + final byId = _bool(step, 'byId'); + // byId factories emit unquoted #key so the Go parser's WaitSelector + // branch (parser.go:846) matches; text factories emit quoted form + // for the WaitAppears/WaitDisappears branch. + final rendered = byId ? '#$target' : '"${_escape(target)}"'; + _buf.writeln('${indent}wait until $rendered ' '${appears ? 'appears' : 'disappears'}'); break; @@ -321,7 +373,11 @@ class ProbeEmitter { final path = _str(step, 'path') ?? ''; final status = _int(step, 'status'); final body = _str(step, 'body') ?? ''; - _buf.writeln('${indent}when the app calls $method $path'); + // Path must be quoted so the lexer doesn't split on `/` and lose + // segments after the first IDENT. Parser test fixture at + // internal/parser/parser_test.go shows quoted form is canonical. + _buf.writeln( + '${indent}when the app calls $method "${_escape(path)}"'); if (body.isNotEmpty) { _buf.writeln( '${' ' * (depth + 1)}respond with $status and body "${_escape(body)}"'); @@ -384,34 +440,57 @@ class ProbeEmitter { _buf.writeln('$indent$verb $dirStr$tail'); } - void _emitSee(DartObject step, int depth) { + void _emitSee(DartObject step, int depth, {bool negated = false}) { final indent = ' ' * depth; - final text = _str(step, 'text') ?? ''; final exactly = step.getField('exactly'); + + // Target: text, id, or rich selector. Mirror the same priority order + // used by the tap-family target resolver. + final selector = step.getField('selector'); + final id = _str(step, 'id'); + final text = _str(step, 'text'); + String target; + if (selector != null && !selector.isNull) { + target = _renderSelector(selector); + } else if (id != null && id.isNotEmpty) { + target = '#$id'; + } else { + target = '"${_escape(text ?? '')}"'; + } + final containing = _str(step, 'containing'); final matching = _str(step, 'matching'); - final stateField = step.getField('state'); - final stateIdx = stateField?.getField('index')?.toIntValue() ?? 0; - // SeeState: 0=none, 1=enabled, 2=disabled, 3=checked, 4=focused - const stateNames = [null, 'enabled', 'disabled', 'checked', 'focused']; - final stateName = - (stateIdx >= 0 && stateIdx < stateNames.length) ? stateNames[stateIdx] : null; + final stateName = _seeStateName(step.getField('state')); final countStr = (exactly == null || exactly.isNull) ? '' : 'exactly ${exactly.toIntValue()} '; - final base = '${countStr}"${_escape(text)}"'; - var line = 'see $base'; - if (stateName != null) { - line = 'see $base is $stateName'; - } + final verb = negated ? "don't see" : 'see'; + + // Compose suffixes additively — state, contains, and matching can ALL + // appear in a single assertion. The Go parser at parser.go:728-769 + // parses them sequentially after the selector, so we emit in the same + // canonical order: contains matching . + final parts = ['$verb $countStr$target'.replaceAll(' ', ' ')]; + if (stateName != null) parts.add('is $stateName'); if (containing != null && containing.isNotEmpty) { - line = 'see $base contains "${_escape(containing)}"'; + parts.add('contains "${_escape(containing)}"'); } if (matching != null && matching.isNotEmpty) { - line = 'see $base matching "${_escape(matching)}"'; + parts.add('matching "${_escape(matching)}"'); } - _buf.writeln('$indent$line'); + _buf.writeln('$indent${parts.join(' ')}'); + } + + /// Maps a SeeState enum DartObject to its parser keyword, or null if + /// SeeState.none. Reads the enum constant identifier rather than trusting + /// the index so reordering the enum doesn't silently corrupt output. + String? _seeStateName(DartObject? stateField) { + if (stateField == null || stateField.isNull) return null; + final name = _enumName(stateField); + // Map DSL enum name → parser keyword. 'none' means no state suffix. + const valid = {'enabled', 'disabled', 'checked', 'focused'}; + return valid.contains(name) ? name : null; } void _emitExamples(DartObject examples) { @@ -516,18 +595,30 @@ class ProbeEmitter { } String _httpMethod(DartObject obj, String field) { - final f = obj.getField(field); - if (f == null) return 'GET'; - final idx = f.getField('index')?.toIntValue() ?? 0; - const names = ['GET', 'POST', 'PUT', 'DELETE']; - return idx >= 0 && idx < names.length ? names[idx] : 'GET'; + final name = _enumName(obj.getField(field)); + // HttpMethod enum: get | post | put | delete → uppercase. + if (name == null || name.isEmpty) return 'GET'; + return name.toUpperCase(); } String _direction(DartObject? d) { if (d == null) return 'down'; - final idx = d.getField('index')?.toIntValue() ?? 0; - const names = ['up', 'down', 'left', 'right']; - return idx >= 0 && idx < names.length ? names[idx] : 'down'; + final name = _enumName(d); + // Direction enum names map 1:1 to ProbeScript keywords. + if (name == null || name.isEmpty) return 'down'; + return name; + } + + /// Returns the Dart enum constant identifier (e.g. "up" for Direction.up). + /// Works on Dart 2.15+ where every enum value has an auto-generated `_name` + /// field. Decouples emitter correctness from enum *declaration order* — + /// reordering a DSL enum no longer silently corrupts emitted ProbeScript. + String? _enumName(DartObject? value) { + if (value == null || value.isNull) return null; + final n = value.getField('_name')?.toStringValue(); + if (n != null) return n; + // Older Dart versions exposed the name via `name`; try that next. + return value.getField('name')?.toStringValue(); } String _ordinal(int n) { diff --git a/probe_gen/pubspec.yaml b/probe_gen/pubspec.yaml index e570f44..c25e13e 100644 --- a/probe_gen/pubspec.yaml +++ b/probe_gen/pubspec.yaml @@ -4,7 +4,7 @@ description: >- flutter_probe_annotation decorators on Flutter classes and emits .probe test files into tests/generated/. Pair with flutter_probe_agent and the probe CLI to run the generated end-to-end tests. -version: 0.9.5 +version: 0.9.6 homepage: https://flutterprobe.dev repository: https://github.com/AlphaWaveSystems/flutter-probe issue_tracker: https://github.com/AlphaWaveSystems/flutter-probe/issues @@ -22,7 +22,7 @@ environment: dependencies: analyzer: ">=8.0.0 <14.0.0" build: ">=3.0.0 <5.0.0" - flutter_probe_annotation: ^0.9.5 + flutter_probe_annotation: ^0.9.6 source_gen: ^4.0.0 dev_dependencies: diff --git a/probe_gen/test/builder_test.dart b/probe_gen/test/builder_test.dart index b13bee9..b9ba406 100644 --- a/probe_gen/test/builder_test.dart +++ b/probe_gen/test/builder_test.dart @@ -59,6 +59,24 @@ void main() { test('emits loops and conditionals with nested bodies', () => runGolden('loops_and_conditionals')); + test('quotes Mock path and emits CallHttp correctly', + () => runGolden('mock_and_call')); + + test('composes See state/contains/matching suffixes additively', + () => runGolden('see_states')); + + test('emits every wait variant including id-based', + () => runGolden('wait_variants')); + + test('emits inline Examples table', + () => runGolden('examples_inline')); + + test('emits composite test with devices, OnDevice blocks, and Sync barriers', + () => runGolden('composite_chat')); + + test('kitchen sink — one of every step + selector + control flow', + () => runGolden('kitchen_sink')); + test('skips files with no FlutterProbe annotations', () async { final annotation = await annotationPackageAssets(); await testBuilder( diff --git a/probe_gen/test/fixtures/composite_chat.dart b/probe_gen/test/fixtures/composite_chat.dart new file mode 100644 index 0000000..1782a25 --- /dev/null +++ b/probe_gen/test/fixtures/composite_chat.dart @@ -0,0 +1,34 @@ +import 'package:flutter_probe_annotation/flutter_probe_annotation.dart'; + +@ProbeCompositeTest( + name: 'alice sends bob a message', + tags: ['composite', 'smoke'], + devices: [ + Device('A', target: 'iPhone 15 Simulator'), + Device('B', target: 'Pixel 9 Emulator'), + ], + body: [ + OnDevice('A', steps: [ + Open(), + Tap(text: 'Sign in as Alice'), + WaitUntil.appears('Inbox'), + ]), + OnDevice('B', steps: [ + Open(), + Tap(text: 'Sign in as Bob'), + WaitUntil.appears('Inbox'), + ]), + Sync('both signed in'), + OnDevice('A', steps: [ + Tap(text: 'New message'), + Type('hello bob'), + Tap(text: 'Send'), + ]), + OnDevice('B', steps: [ + WaitUntil.appears('hello bob'), + See('hello bob'), + ]), + Sync('message delivered'), + ], +) +class ChatComposite {} diff --git a/probe_gen/test/fixtures/composite_chat.probe.golden b/probe_gen/test/fixtures/composite_chat.probe.golden new file mode 100644 index 0000000..8a9b106 --- /dev/null +++ b/probe_gen/test/fixtures/composite_chat.probe.golden @@ -0,0 +1,26 @@ +# Generated by flutter_probe_gen — do not edit by hand. +# Source: lib/composite_chat.dart + +composite test "alice sends bob a message" + @composite @smoke + devices + A: iPhone 15 Simulator + B: Pixel 9 Emulator + A: + open the app + tap "Sign in as Alice" + wait until "Inbox" appears + B: + open the app + tap "Sign in as Bob" + wait until "Inbox" appears + sync "both signed in" + A: + tap "New message" + type "hello bob" + tap "Send" + B: + wait until "hello bob" appears + see "hello bob" + sync "message delivered" + diff --git a/probe_gen/test/fixtures/examples_inline.dart b/probe_gen/test/fixtures/examples_inline.dart new file mode 100644 index 0000000..d9ad946 --- /dev/null +++ b/probe_gen/test/fixtures/examples_inline.dart @@ -0,0 +1,20 @@ +import 'package:flutter_probe_annotation/flutter_probe_annotation.dart'; + +@ProbeSuite( + tests: [ + ProbeTest('login with ', steps: [ + Open(), + Type('', into: Field(id: 'email')), + Type('', into: Field(id: 'password')), + Tap(text: 'Sign In'), + See(''), + ], examples: Examples( + headers: ['email', 'password', 'result'], + rows: [ + ['alice@test.com', 'hunter2', 'Dashboard'], + ['bob@test.com', 'wrong', 'Invalid credentials'], + ], + )), + ], +) +class InlineExamplesScreen {} diff --git a/probe_gen/test/fixtures/examples_inline.probe.golden b/probe_gen/test/fixtures/examples_inline.probe.golden new file mode 100644 index 0000000..02d6e0e --- /dev/null +++ b/probe_gen/test/fixtures/examples_inline.probe.golden @@ -0,0 +1,14 @@ +# Generated by flutter_probe_gen — do not edit by hand. +# Source: lib/examples_inline.dart + +test "login with " + open the app + type "" into #email + type "" into #password + tap "Sign In" + see "" + with examples: + | email | password | result | + | alice@test.com | hunter2 | Dashboard | + | bob@test.com | wrong | Invalid credentials | + diff --git a/probe_gen/test/fixtures/kitchen_sink.dart b/probe_gen/test/fixtures/kitchen_sink.dart new file mode 100644 index 0000000..d6315ca --- /dev/null +++ b/probe_gen/test/fixtures/kitchen_sink.dart @@ -0,0 +1,66 @@ +import 'package:flutter_probe_annotation/flutter_probe_annotation.dart'; + +@ProbeSuite( + tests: [ + ProbeTest('kitchen sink — one of every step type', steps: [ + // App lifecycle + Open(), + OpenLink('https://example.com/foo'), + Close(), + Restart(), + Kill(), + ClearAppData(), + // Tap-family + Tap(text: 'Login'), + Tap(id: 'submit'), + Tap(text: 'Cookies', ifVisible: true), + DoubleTap(text: 'Map'), + LongPress(id: 'thumbnail'), + GoBack(), + // Text input + Type('alice@example.com', into: Field(id: 'email')), + Type('hunter2', into: Field(text: 'Password'), ifVisible: true), + Clear(id: 'search'), + // Motion + Swipe.up(), + Swipe.down(on: IdSel('list')), + Scroll.up(), + Scroll.right(on: TextSel('Container')), + Drag(from: IdSel('a'), to: IdSel('b')), + Rotate.landscape(), + Toggle('Notifications'), + Shake(), + // Permissions + AllowPermission('camera'), + DenyPermission('contacts'), + GrantAllPermissions(), + RevokeAllPermissions(), + // Clipboard / device + CopyToClipboard('hello world'), + PasteFromClipboard(), + SetLocation(37.7749, -122.4194), + VerifyExternalBrowser(), + // Diagnostics + TakeScreenshot('snap'), + CompareScreenshot('snap'), + DumpWidgetTree(), + SaveLogs(), + Pause(), + Log('breadcrumb here'), + // Variables + Store('xyz', as: 'token'), + // Control flow + If('Onboarding', then: [Tap(text: 'Skip')], otherwise: [Tap(text: 'Continue')]), + Repeat(2, body: [Swipe.up(), WaitFor.duration(0.25)]), + // Selector zoo via See + See.selector(TypeSel('ElevatedButton')), + See.selector(InContainer('Email', container: 'LoginForm')), + See.selector(Above('Title', anchor: 'Subtitle')), + See.selector(LeftOf('Item1', anchor: 'Item2')), + See.selector(RightOf('Item3', anchor: 'Item2')), + // Dart escape + RunDart('debugPrint("hi");\nfinal x = 42;'), + ]), + ], +) +class KitchenSinkScreen {} diff --git a/probe_gen/test/fixtures/kitchen_sink.probe.golden b/probe_gen/test/fixtures/kitchen_sink.probe.golden new file mode 100644 index 0000000..4559afa --- /dev/null +++ b/probe_gen/test/fixtures/kitchen_sink.probe.golden @@ -0,0 +1,58 @@ +# Generated by flutter_probe_gen — do not edit by hand. +# Source: lib/kitchen_sink.dart + +test "kitchen sink — one of every step type" + open the app + open link "https://example.com/foo" + close the app + restart the app + kill the app + clear app data + tap "Login" + tap #submit + tap "Cookies" if visible + double tap "Map" + long press #thumbnail + go back + type "alice@example.com" into #email + type "hunter2" into "Password" if visible + clear #search + swipe up + swipe down #list + scroll up + scroll right "Container" + drag #a to #b + rotate landscape + toggle "Notifications" + shake + allow permission "camera" + deny permission "contacts" + grant all permissions + revoke all permissions + copy "hello world" to clipboard + paste from clipboard + set location 37.7749, -122.4194 + verify external browser opened + take screenshot "snap" + compare screenshot "snap" + dump widget tree + save logs + pause + log "breadcrumb here" + store "xyz" as token + if "Onboarding" appears + tap "Skip" + otherwise + tap "Continue" + repeat 2 times + swipe up + wait 0.25 seconds + see ElevatedButton + see "Email" in "LoginForm" + see "Title" above "Subtitle" + see "Item1" left of "Item2" + see "Item3" right of "Item2" + run dart: + debugPrint("hi"); + final x = 42; + diff --git a/probe_gen/test/fixtures/mock_and_call.dart b/probe_gen/test/fixtures/mock_and_call.dart new file mode 100644 index 0000000..01218ba --- /dev/null +++ b/probe_gen/test/fixtures/mock_and_call.dart @@ -0,0 +1,17 @@ +import 'package:flutter_probe_annotation/flutter_probe_annotation.dart'; + +@ProbeSuite( + tests: [ + ProbeTest('mocks api and calls webhook', steps: [ + Mock(method: HttpMethod.get, path: '/api/products', status: 200, + body: '[{"id":1,"name":"Widget"}]'), + Mock(method: HttpMethod.post, path: '/api/orders', status: 201), + CallHttp(method: HttpMethod.post, url: 'https://example.com/hook', + body: '{"event":"order_created"}'), + CallHttp(method: HttpMethod.get, url: 'https://example.com/health'), + Open(), + See('Welcome'), + ]), + ], +) +class MockAndCallScreen {} diff --git a/probe_gen/test/fixtures/mock_and_call.probe.golden b/probe_gen/test/fixtures/mock_and_call.probe.golden new file mode 100644 index 0000000..b42e399 --- /dev/null +++ b/probe_gen/test/fixtures/mock_and_call.probe.golden @@ -0,0 +1,13 @@ +# Generated by flutter_probe_gen — do not edit by hand. +# Source: lib/mock_and_call.dart + +test "mocks api and calls webhook" + when the app calls GET "/api/products" + respond with 200 and body "[{\"id\":1,\"name\":\"Widget\"}]" + when the app calls POST "/api/orders" + respond with 201 + call POST "https://example.com/hook" with body "{\"event\":\"order_created\"}" + call GET "https://example.com/health" + open the app + see "Welcome" + diff --git a/probe_gen/test/fixtures/see_states.dart b/probe_gen/test/fixtures/see_states.dart new file mode 100644 index 0000000..b4b7b79 --- /dev/null +++ b/probe_gen/test/fixtures/see_states.dart @@ -0,0 +1,25 @@ +import 'package:flutter_probe_annotation/flutter_probe_annotation.dart'; + +@ProbeSuite( + tests: [ + ProbeTest('exhaustive see assertions', steps: [ + See('Welcome'), + See('Sign In', state: SeeState.enabled), + See('Submit', state: SeeState.disabled), + See('Agree to terms', state: SeeState.checked), + See('email', state: SeeState.focused), + See('Logout', containing: 'out'), + See('123-456-7890', matching: r'^\d{3}-\d{3}-\d{4}$'), + // Combined: state + containing in the same assertion (the bug fix). + See('email field', state: SeeState.enabled, containing: 'email'), + See('Item', exactly: 5), + DontSee('Error'), + DontSee.id('error_banner'), + // Selector forms — id and rich selector. + See.id('password_field', state: SeeState.focused), + See.selector(Ordinal(2, 'List Item')), + See.selector(Below('Subtitle', anchor: 'Title')), + ]), + ], +) +class SeeStatesScreen {} diff --git a/probe_gen/test/fixtures/see_states.probe.golden b/probe_gen/test/fixtures/see_states.probe.golden new file mode 100644 index 0000000..75d8f95 --- /dev/null +++ b/probe_gen/test/fixtures/see_states.probe.golden @@ -0,0 +1,19 @@ +# Generated by flutter_probe_gen — do not edit by hand. +# Source: lib/see_states.dart + +test "exhaustive see assertions" + see "Welcome" + see "Sign In" is enabled + see "Submit" is disabled + see "Agree to terms" is checked + see "email" is focused + see "Logout" contains "out" + see "123-456-7890" matching "^\\d{3}-\\d{3}-\\d{4}$" + see "email field" is enabled contains "email" + see exactly 5 "Item" + don't see "Error" + don't see #error_banner + see #password_field is focused + see 2nd "List Item" + see "Subtitle" below "Title" + diff --git a/probe_gen/test/fixtures/wait_variants.dart b/probe_gen/test/fixtures/wait_variants.dart new file mode 100644 index 0000000..fa9ccfa --- /dev/null +++ b/probe_gen/test/fixtures/wait_variants.dart @@ -0,0 +1,19 @@ +import 'package:flutter_probe_annotation/flutter_probe_annotation.dart'; + +@ProbeSuite( + tests: [ + ProbeTest('every wait variant', steps: [ + Open(), + WaitFor.duration(1.5), + WaitForPageLoad(), + WaitForNetworkIdle(), + WaitForAnimations(), + WaitUntil.appears('Dashboard'), + WaitUntil.disappears('Loading'), + WaitUntil.idAppears('login_form'), + WaitUntil.idDisappears('spinner'), + See('Dashboard'), + ]), + ], +) +class WaitVariantsScreen {} diff --git a/probe_gen/test/fixtures/wait_variants.probe.golden b/probe_gen/test/fixtures/wait_variants.probe.golden new file mode 100644 index 0000000..00482f6 --- /dev/null +++ b/probe_gen/test/fixtures/wait_variants.probe.golden @@ -0,0 +1,15 @@ +# Generated by flutter_probe_gen — do not edit by hand. +# Source: lib/wait_variants.dart + +test "every wait variant" + open the app + wait 1.5 seconds + wait for the page to load + wait until network is idle + wait for animations to end + wait until "Dashboard" appears + wait until "Loading" disappears + wait until #login_form appears + wait until #spinner disappears + see "Dashboard" + diff --git a/vscode/package.json b/vscode/package.json index 2698e25..ab48ac7 100644 --- a/vscode/package.json +++ b/vscode/package.json @@ -2,7 +2,7 @@ "name": "flutterprobe", "displayName": "FlutterProbe", "description": "High-performance E2E testing for Flutter apps — ProbeScript language support, local & cloud device testing (BrowserStack, Sauce Labs, AWS Device Farm, Firebase Test Lab, LambdaTest), visual regression, test recording, and Studio integration", - "version": "0.9.5", + "version": "0.9.6", "publisher": "flutterprobe", "icon": "resources/probe-icon.png", "engines": { "vscode": "^1.85.0" }, diff --git a/website/astro.config.mjs b/website/astro.config.mjs index b238c68..0e50778 100644 --- a/website/astro.config.mjs +++ b/website/astro.config.mjs @@ -32,6 +32,7 @@ export default defineConfig({ { label: 'Recipes', slug: 'probescript/recipes' }, { label: 'Data-Driven Tests', slug: 'probescript/data-driven' }, { label: 'Hooks', slug: 'probescript/hooks' }, + { label: 'Annotation-driven Tests', slug: 'probescript/annotations' }, { label: 'Dictionary', slug: 'probescript/dictionary' }, ], }, diff --git a/website/src/content/docs/probescript/annotations.md b/website/src/content/docs/probescript/annotations.md new file mode 100644 index 0000000..bb5196b --- /dev/null +++ b/website/src/content/docs/probescript/annotations.md @@ -0,0 +1,251 @@ +--- +title: Annotation-driven Tests +description: Declare ProbeScript tests as Dart annotations on your Flutter widget classes — type-checked by the compiler, generated to .probe at build time. +--- + +Annotations let you keep test definitions next to the widget code they exercise. Two Dart packages handle the loop: + +- **[`flutter_probe_annotation`](https://pub.dev/packages/flutter_probe_annotation)** — `@ProbeSuite`, `@ProbeTest`, `@ProbeRecipe`, `@ProbeCompositeTest` decorators plus a fully type-checked step DSL. +- **[`flutter_probe_gen`](https://pub.dev/packages/flutter_probe_gen)** — `build_runner` builder that reads the annotations and emits matching `.probe` files into `tests/generated/`. + +A renamed button or translated label no longer silently breaks tests — selectors live in the same file as the widget that owns them, and every step is type-checked by `flutter analyze` before it ever runs. + +## Install + +```yaml +# pubspec.yaml +dependencies: + flutter_probe_annotation: ^0.9.6 + flutter_probe_agent: ^0.9.6 + +dev_dependencies: + flutter_probe_gen: ^0.9.6 + build_runner: ^2.15.0 +``` + +Run the builder once per change: + +```bash +dart run build_runner build +``` + +A `.probe` file is written under `tests/generated/` for every annotated Dart file: + +``` +lib/screens/login_screen.dart → tests/generated/screens/login_screen.probe +``` + +Then run them with the regular CLI: + +```bash +probe test tests/ +``` + +## Annotations + +### `@ProbeSuite` + +Top-level annotation on any class. Groups tests, hooks, and recipes that share setup logic. + +```dart +import 'package:flutter_probe_annotation/flutter_probe_annotation.dart'; + +@ProbeSuite( + name: 'Login', + beforeEach: [Open()], + tests: [ + ProbeTest('user can log in', tags: ['smoke'], steps: [ + Tap(id: 'email_field'), + Type('alice@example.com'), + Tap(id: 'password_field'), + Type('hunter2'), + Tap(text: 'Sign In'), + WaitUntil.appears('Dashboard'), + See('Dashboard'), + ]), + ], +) +class LoginScreen extends StatelessWidget { /* … */ } +``` + +| Field | Emits | +|---|---| +| `tests` | one `test "name"` block per `ProbeTest` | +| `beforeEach` / `afterEach` | `before each test` / `after each test` | +| `beforeAll` / `afterAll` | `before all tests` / `after all tests` | +| `onFailure` | `on failure` hook | +| `recipes` | one `recipe "name"` block per `ProbeRecipe` | + +### `@ProbeTest` + +Single test — used inside `ProbeSuite.tests` or as a standalone top-level annotation. + +| Field | Emits | +|---|---| +| `name` | `test "name"` | +| `tags: ['smoke']` | `@smoke` line | +| `steps` | indented test body | +| `examples` | `with examples:` table | + +### `@ProbeRecipe` + +Reusable recipe with named parameters. Reference parameters as `` inside any string field. + +```dart +ProbeRecipe('sign in', params: ['email', 'password'], steps: [ + Tap(id: 'email_field'), + Type(''), + Tap(id: 'password_field'), + Type(''), + Tap(text: 'Sign In'), +]) +``` + +Invoke from a test with `RecipeStep('sign in', args: ['a@b.com', 'pw'])`. + +### `@ProbeCompositeTest` (v0.9.6+) + +Declares a multi-device composite test. Devices are listed by alias; per-device step groups use `OnDevice`, and `Sync` barriers force all devices to reach a checkpoint together. + +```dart +@ProbeCompositeTest( + name: 'alice sends bob a message', + tags: ['composite', 'smoke'], + devices: [ + Device('A', target: 'iPhone 15 Simulator'), + Device('B', target: 'Pixel 9 Emulator'), + ], + body: [ + OnDevice('A', steps: [Open(), Tap(text: 'Sign in as Alice')]), + OnDevice('B', steps: [Open(), Tap(text: 'Sign in as Bob')]), + Sync('both signed in'), + OnDevice('A', steps: [ + Tap(text: 'New message'), + Type('hello bob'), + Tap(text: 'Send'), + ]), + OnDevice('B', steps: [ + WaitUntil.appears('hello bob'), + See('hello bob'), + ]), + ], +) +class ChatComposite {} +``` + +The emitted `.probe` block uses the standard `composite test` / `devices` / `sync` syntax. See the [composite test guide](/probescript/syntax/#composite-tests) for runtime details. + +## Step DSL — full coverage + +All 31 ProbeScript actions have a matching `const` Dart class. Common ones: + +| Class | Emits | +|---|---| +| `Open()` / `OpenLink(url)` / `Close()` | `open the app` / `open link "url"` / `close the app` | +| `Restart()` / `Kill()` / `ClearAppData()` | corresponding lifecycle action | +| `Tap(id: 'login')` / `Tap(text: 'Sign In')` | `tap #login` / `tap "Sign In"` | +| `Tap(id: 'x', ifVisible: true)` | `tap #x if visible` | +| `DoubleTap` / `LongPress` / `GoBack()` | as named | +| `Type('hello', into: Field(id: 'msg'))` | `type "hello" into #msg` | +| `Clear(id: 'x')` | `clear #x` | +| `Swipe.up()` / `Scroll.down(on: …)` | `swipe up` / `scroll down …` | +| `Drag(from: …, to: …)` | `drag "from" to "to"` | +| `Rotate.landscape()` / `Toggle('switch')` / `Shake()` | as named | +| `AllowPermission('camera')` / `DenyPermission('mic')` | `allow permission "camera"` | +| `GrantAllPermissions()` / `RevokeAllPermissions()` | as named | +| `CopyToClipboard('x')` / `PasteFromClipboard()` | clipboard ops | +| `SetLocation(lat, lng)` / `VerifyExternalBrowser()` | as named | +| `TakeScreenshot('name')` / `CompareScreenshot('name')` | screenshot ops | +| `DumpWidgetTree()` / `SaveLogs()` / `Pause()` / `Log('msg')` | as named | +| `Store('value', as: 'var')` | `store "value" as var` | +| `See('X')` / `See('X', state: SeeState.enabled)` / `See('X', exactly: 2)` | `see "X"` and variants | +| `See.id('x', state: SeeState.focused)` (v0.9.6+) | `see #x is focused` | +| `See.selector(Ordinal(2, 'Item'))` (v0.9.6+) | `see 2nd "Item"` | +| `DontSee('X')` / `DontSee.id('x')` (v0.9.6+) | `don't see "X"` / `don't see #x` | +| `WaitFor.duration(N)` | `wait N seconds` | +| `WaitUntil.appears('X')` / `.disappears('X')` | `wait until "X" appears` etc. | +| `WaitUntil.idAppears('x')` (v0.9.6+) | `wait until #x appears` | +| `WaitForPageLoad()` / `WaitForNetworkIdle()` / `WaitForAnimations()` | as named | +| `If('cond', then: [...], otherwise: [...])` | `if "cond" appears` block | +| `Repeat(N, body: [...])` | `repeat N times` block | +| `RunDart('print("hi");')` | `run dart:` block | +| `Mock(method: HttpMethod.get, path: '/x', status: 200, body: '{…}')` | `when the app calls GET "/x"` block | +| `CallHttp(method: HttpMethod.post, url: '…', body: '…')` | `call POST "…" with body "…"` | +| `RecipeStep('name', args: [...])` | recipe invocation | + +### Selectors + +```dart +// Convenience (most common) +Tap(text: 'Sign In') +Tap(id: 'login_button') + +// Explicit +Tap(selector: TextSel('Sign In')) +Tap(selector: IdSel('login_button')) +Tap(selector: TypeSel('ElevatedButton')) +Tap(selector: Ordinal(2, 'Item', container: 'List')) +Tap(selector: Below('Subtitle', anchor: 'Title')) +Tap(selector: Above('a', anchor: 'b')) +Tap(selector: LeftOf('a', anchor: 'b')) +Tap(selector: RightOf('a', anchor: 'b')) +Tap(selector: InContainer('Email', container: 'LoginForm')) +``` + +## See / DontSee — composable assertions (v0.9.6+) + +`state`, `containing`, and `matching` can all coexist on a single `See`: + +```dart +See('email field', state: SeeState.enabled, containing: 'email') +// → see "email field" is enabled contains "email" +``` + +`See.id` / `See.selector` target by `ValueKey` or rich selector: + +```dart +See.id('password_field', state: SeeState.focused) +// → see #password_field is focused + +See.selector(Below('Subtitle', anchor: 'Title')) +// → see "Subtitle" below "Title" +``` + +Same factories exist for `DontSee`. + +## Output layout + +Each annotated source file produces a single `.probe` in `tests/generated/`, preserving directory structure: + +``` +lib/screens/login.dart → tests/generated/screens/login.probe +lib/features/chat/chat.dart → tests/generated/features/chat/chat.probe +``` + +The generated file starts with a `do not edit` header that includes the source path. Run them like any other tests: + +```bash +probe test tests/ +``` + +## Should `tests/generated/` be committed? + +Both workflows are valid: + +- **Commit it** — review changes in PR like any other code, no need to run `build_runner` in CI before `probe test`. Add a CI step that fails if the builder produces diffs (catches forgotten regenerations). +- **Gitignore it** — single source of truth lives in the Dart code; CI runs `dart run build_runner build` before `probe test`. + +Pick whichever fits your team. + +## Cross-language validation + +Every fixture in `flutter_probe_gen`'s test suite is round-tripped through the Go-side ProbeScript parser as part of CI (`internal/parser/golden_integration_test.go`). If the Dart emitter ever produces output the runtime can't parse, the Go test fails and the release is blocked — bugs are caught in CI, not at user runtime. + +## Limitations + +A few step types are exposed in the DSL but currently **not supported** by the runtime — using them in your tests will produce a "command not implemented" error at runtime: + +- `Press('home')` / `Press('back')` — marked `@Deprecated` in v0.9.6. Will be enabled once the Go parser and Dart agent support platform key presses. +- `Pinch(zoomIn: true)` — same status. + +Use `GoBack()` (which is fully supported) in place of `Press('back')`. diff --git a/website/src/content/docs/tools/mcp.md b/website/src/content/docs/tools/mcp.md index b2c078e..12ff0f4 100644 --- a/website/src/content/docs/tools/mcp.md +++ b/website/src/content/docs/tools/mcp.md @@ -238,7 +238,7 @@ echo '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' | probe-mcp Expected response: ```json -{"jsonrpc":"2.0","id":1,"result":{"capabilities":{"tools":{}},"protocolVersion":"2024-11-05","serverInfo":{"name":"probe-mcp","version":"0.9.5"}}} +{"jsonrpc":"2.0","id":1,"result":{"capabilities":{"tools":{}},"protocolVersion":"2024-11-05","serverInfo":{"name":"probe-mcp","version":"0.9.6"}}} ``` List all available tools: