Add `dotnet test` integration tests for all Apple platforms by jonathanpeppers · Pull Request #25320 · dotnet/macios

jonathanpeppers · 2026-05-04T21:58:14Z

This PR adds end-to-end dotnet test integration tests for all Apple platforms (iOS, tvOS, Mac Catalyst, macOS) using the MTP (Microsoft Testing Platform) test templates.

Changes

tests/dotnet/UnitTests/DotNetTestTest.cs: New parameterized test that creates a project from dotnet new <template>, replaces the generated tests with a single passing test, and runs dotnet test.
tests/common/DotNet.cs: Added test verb support to DotNet.Execute.

The test reads the MLAUNCH_PATH environment variable to override the mlaunch binary for mobile platforms (iOS/tvOS), enabling testing with the MTP-enabled mlaunch build.

Screenshot

Create DotNetTestTest that runs 'dotnet test' on projects created from the iostest, tvostest, maccatalysttest, and macostest templates. This verifies the MTP (Microsoft Testing Platform) pipeline works end-to-end, including the --server/--dotnet-test-pipe argument passthrough via RunArguments. Also add 'test' as a supported verb in DotNet.Execute. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

macOS and Mac Catalyst don't use mlaunch (they use 'open' via Desktop.targets), so MTP support requires a different approach. Comment out those test cases for now. Also skip the /v:diag and /consoleloggerparameters workaround args when the verb is 'test' to prevent them leaking through to the MTP test runner via RunArguments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Update mlaunch NuGet package to include MTP (Microsoft Testing Platform) support for iOS and tvOS simulator test execution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The NUnit runner process sets MSBuildSDKsPath and MSBUILD_EXE_PATH which interfere with the child dotnet process SDK resolution. Every DotNet.Execute call clears these; we must do the same. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

vs-mobiletools-engineering-service2 · 2026-05-15T19:41:14Z

✅ [CI Build #55f6051] Build passed (Build packages) ✅

Pipeline on Agent
Hash: 55f6051a0191cd984dab8556444b58e7b1170a0a [PR build]

vs-mobiletools-engineering-service2 · 2026-05-15T19:44:48Z

✅ [CI Build #55f6051] Build passed (Detect API changes) ✅

Pipeline on Agent
Hash: 55f6051a0191cd984dab8556444b58e7b1170a0a [PR build]

vs-mobiletools-engineering-service2 · 2026-05-15T19:56:55Z

✅ API diff for current PR / commit

NET (empty diffs)

iOS: vsdrops gist ( No breaking changes )
tvOS: vsdrops gist ( No breaking changes )
MacCatalyst: vsdrops gist ( No breaking changes )
macOS: vsdrops gist ( No breaking changes )

✅ API diff vs stable

NET (empty diffs)

iOS: vsdrops gist ( No breaking changes )
tvOS: vsdrops gist ( No breaking changes )
MacCatalyst: vsdrops gist ( No breaking changes )
macOS: vsdrops gist ( No breaking changes )

ℹ️ Generator diff

Generator Diff: vsdrops (html) vsdrops (raw diff) gist (raw diff) - Please review changes)

Pipeline on Agent
Hash: 55f6051a0191cd984dab8556444b58e7b1170a0a [PR build]

vs-mobiletools-engineering-service2 · 2026-05-15T20:05:36Z

✅ [CI Build #55f6051] Build passed (Build macOS tests) ✅

Pipeline on Agent
Hash: 55f6051a0191cd984dab8556444b58e7b1170a0a [PR build]

vs-mobiletools-engineering-service2 · 2026-05-15T21:36:38Z

🔥 [CI Build #55f6051] Test results 🔥

Test results

❌ Tests failed on VSTS: test results

0 tests crashed, 10 tests failed, 171 tests passed.

Failures

❌ dotnettests tests (iOS)

1 tests failed, 0 tests passed.

Failed tests

DotNet tests: Failed (Execution failed with exit code 1)
- Xamarin.Tests.DotNetTestTest.DotNetTest(iOS,"iostest"): 'dotnet test' failed with exit code 1.
  Output:
  Running the ComputeRunArguments target to discover run commands failed for this...

Html Report (VSDrops) Download

❌ dotnettests tests (tvOS)

1 tests failed, 0 tests passed.

Failed tests

DotNet tests: Failed (Execution failed with exit code 1)
- Xamarin.Tests.DotNetTestTest.DotNetTest(TVOS,"tvostest"): 'dotnet test' failed with exit code 1.
  Output:
  Running the ComputeRunArguments target to discover run commands failed for this...

Html Report (VSDrops) Download

❌ interdependent-binding-projects tests

2 tests failed, 2 tests passed.

Failed tests

interdependent-binding-projects/iOS - simulator/Debug: LaunchTimedOut
interdependent-binding-projects/tvOS - simulator/Debug: LaunchTimedOut

Html Report (VSDrops) Download

❌ introspection tests

6 tests failed, 3 tests passed.

Failed tests

introspection/iOS - simulator/Debug: LaunchTimedOut
introspection/iOS - simulator/CoreCLR: LaunchTimedOut
introspection/iOS - simulator/Debug (ARM64): LaunchTimedOut
introspection/tvOS - simulator/Debug: LaunchTimedOut
introspection/tvOS - simulator/CoreCLR: LaunchTimedOut
introspection/tvOS - simulator/Debug (ARM64): LaunchTimedOut

Html Report (VSDrops) Download

Successes

✅ cecil: All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (MacCatalyst): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (macOS): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (Multiple platforms): All 1 tests passed. Html Report (VSDrops) Download
✅ framework: All 2 tests passed. Html Report (VSDrops) Download
✅ fsharp: All 4 tests passed. Html Report (VSDrops) Download
✅ generator: All 5 tests passed. Html Report (VSDrops) Download
✅ linker: All 44 tests passed. Html Report (VSDrops) Download
✅ monotouch (iOS): All 19 tests passed. Html Report (VSDrops) Download
✅ monotouch (MacCatalyst): All 25 tests passed. Html Report (VSDrops) Download
✅ monotouch (macOS): All 18 tests passed. Html Report (VSDrops) Download
✅ monotouch (tvOS): All 19 tests passed. Html Report (VSDrops) Download
✅ msbuild: All 2 tests passed. Html Report (VSDrops) Download
✅ sharpie: All 1 tests passed. Html Report (VSDrops) Download
✅ windows: All 3 tests passed. Html Report (VSDrops) Download
✅ xcframework: All 4 tests passed. Html Report (VSDrops) Download
✅ xtro: All 1 tests passed. Html Report (VSDrops) Download

macOS tests

✅ Tests on macOS Sonoma (14): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Sequoia (15): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Tahoe (26): All 5 tests passed. Html Report (VSDrops) Download

Linux Build Verification

✅ Linux build succeeded

Pipeline on Agent
Hash: 55f6051a0191cd984dab8556444b58e7b1170a0a [PR build]

jonathanpeppers changed the base branch from main to net11.0 May 4, 2026 21:58

This comment has been minimized.

Sign in to view

rolfbjarne mentioned this pull request May 11, 2026

[CI Postmortem] Flaky: TestNSUrlSessionEphemeralHandlerCookies on iOS/MacCatalyst #25384

Closed

jonathanpeppers and others added 2 commits May 12, 2026 12:33

jonathanpeppers force-pushed the jonathanpeppers/add-mlaunch-mtp-tests branch 2 times, most recently from 32ce330 to 5c37575 Compare May 12, 2026 17:34

Bump Microsoft.Tools.Mlaunch to 1.1.115

4c9418b

Update mlaunch NuGet package to include MTP (Microsoft Testing Platform) support for iOS and tvOS simulator test execution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

jonathanpeppers force-pushed the jonathanpeppers/add-mlaunch-mtp-tests branch from 5c37575 to 4c9418b Compare May 12, 2026 17:35

This comment has been minimized.

Sign in to view

Conversation

jonathanpeppers commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Screenshot

Related

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

vs-mobiletools-engineering-service2 commented May 15, 2026

✅ [CI Build #55f6051] Build passed (Build packages) ✅

Uh oh!

vs-mobiletools-engineering-service2 commented May 15, 2026

✅ [CI Build #55f6051] Build passed (Detect API changes) ✅

Uh oh!

This comment has been minimized.

This comment has been minimized.

vs-mobiletools-engineering-service2 commented May 15, 2026

✅ API diff for current PR / commit

✅ API diff vs stable

ℹ️ Generator diff

Uh oh!

vs-mobiletools-engineering-service2 commented May 15, 2026

✅ [CI Build #55f6051] Build passed (Build macOS tests) ✅

Uh oh!

vs-mobiletools-engineering-service2 commented May 15, 2026

🔥 [CI Build #55f6051] Test results 🔥

Test results

Failures

❌ dotnettests tests (iOS)

Failed tests

❌ dotnettests tests (tvOS)

Failed tests

❌ interdependent-binding-projects tests

Failed tests

❌ introspection tests

Failed tests

Successes

jonathanpeppers commented May 4, 2026 •

edited

Loading