Feature/mtp test adapter 2803 - #3229
Conversation
Added a new guide for running benchmarks with Microsoft.Testing.Platform (MTP), covering setup, usage, and caveats. Updated the table of contents to include the new page and added a note to the VSTest docs about the MTP adapter option.
Added InternalsVisibleTo attribute in AssemblyInfo.cs to expose internal members to the BenchmarkDotNet.TestAdapter.TestingPlatform assembly, ensuring it uses the same public key as other related assemblies.
Deleted the internal static method GetUnrandomizedJobDisplayInfo from BenchmarkCaseExtensions.cs. This method handled normalization of job display info by removing randomness from job IDs for consistent benchmark referencing. No other code changes were made.
Introduced BenchmarkCaseIdentityExtensions with GetUnrandomizedJobDisplayInfo to normalize Job DisplayInfo by removing random ID components. This ensures consistent benchmark identification across processes for test adapters.
Introduce GetBenchmarksFromAssembly to extract benchmarks from an already loaded Assembly. Refactor existing logic to use this method, improving code reuse and enabling benchmark retrieval from both loaded assemblies and file paths.
Add MSBuild props to enable TestingPlatform integration, set defaults for `dotnet test` compatibility, disable parallel TFM runs, and auto-register BenchmarkDotNet builder hook.
Introduced AsyncWorkQueue, an internal sealed class in BenchmarkDotNet.TestAdapter.TestingPlatform. It enables ordered, thread-safe queuing of asynchronous work items, allowing synchronous producers and asynchronous consumers. Utilizes ConcurrentQueue and SemaphoreSlim, supports completion signaling, and implements IDisposable for resource cleanup.
Created a new .csproj targeting netstandard2.0 for the TestingPlatform adapter. Configured project metadata, packaging, and references. Integrated Microsoft.Testing.Platform.MSBuild and BenchmarkDotNet, and linked shared source files for benchmark enumeration. Set IsTestingPlatformApplication to false to avoid test app behavior.
Introduced BenchmarkDotNetExtension class implementing IExtension to provide extension metadata and enablement status for Microsoft.Testing.Platform integration.
Introduced BenchmarkEventProcessor to process BenchmarkDotNet events and translate them into test node updates for the testing platform. Handles validation errors, build results, benchmark execution, and ensures all benchmarks have published results. Includes logic for error aggregation, output formatting, and timing information.
Introduce BenchmarkTestFramework to integrate BenchmarkDotNet with Microsoft.Testing.Platform, enabling benchmark discovery and execution. Implements session management, filtering, event processing, and output routing. Handles test node updates and cancellation, with support for experimental platform features.
Introduced the internal sealed class BenchmarkTestNode to encapsulate immutable BenchmarkCase data for Microsoft.Testing.Platform integration. This includes stable UID generation, display name and path construction, property management, and support for test filtering and message bus conversion.
Introduced OutputDeviceLogger class implementing ILogger to forward BenchmarkDotNet logs to the platform output device. Handles log kinds, buffers lines, and asynchronously displays output to ensure build progress and results are visible in test run output.
Introduce TestApplicationBuilderExtensions with AddBenchmarkDotNet methods for integrating BenchmarkDotNet benchmarks into Microsoft.Testing.Platform. Includes overloads for entry assembly and specific assemblies, null checks, test framework registration, and tree node filter service support.
Introduced a static TestingPlatformBuilderHook class in the BenchmarkDotNet.TestAdapter.TestingPlatform namespace. This class provides an AddExtensions method to register BenchmarkDotNet with the test application builder, intended for use by generated code and hidden from IntelliSense.
Added a "test" section to global.json to specify "Microsoft.Testing.Platform" as the test runner. This configures the project to use the designated testing platform.
Introduce BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj targeting net10.0 as an executable. The project includes assembly metadata, references BenchmarkDotNet.TestAdapter.TestingPlatform, manually imports its build props, and uses shared common.props and common.targets for build configuration.
Introduced SampleBenchmarks class in BenchmarkDotNet.IntegrationTests.TestingPlatform. Defines Add and Multiply benchmarks with parameterized Size, categorized as "Fast" and "Slow". Uses a custom FastConfig to run benchmarks in-process with a single dry iteration for quick end-to-end testing.
Added BenchmarkDotNet.TestAdapter.TestingPlatform and BenchmarkDotNet.IntegrationTests.TestingPlatform projects to the solution. Updated MonoBenchmarks and SharedDiagnosers integration test projects to only build for the solution in Debug configuration.
Explicitly set BenchmarkDotNet.TestAdapter.TestingPlatform and BenchmarkDotNet.IntegrationTests.TestingPlatform to not build in the Debug configuration by adding <Build Solution="Debug|*" Project="false" /> in BenchmarkDotNet.slnx. No other changes made.
|
Let's name it
Why? We run tests in Release configuration. |
| /// The job is always part of the uid, otherwise two cases of the same benchmark that only differ by job would | ||
| /// collide. The parameters are already part of the method name. | ||
| /// </remarks> | ||
| public static string GetUid(BenchmarkCase benchmarkCase) |
There was a problem hiding this comment.
I though GetUid logics should be implemented on BenchmarkDotNet core project side.
Because --filter-uid option is useful for normal benchmark exe project without MTP.
I've implemented MSTest based UID generation logics on #3227.
Is it able to confirm these logics can be shared with TestAdapter?
There was a problem hiding this comment.
This has been noted and taken into consideration. I have done the fix
There was a problem hiding this comment.
#3227 is merged to master.
So GUID based UID generator is available.
|
|
||
| var properties = new List<IProperty> | ||
| { | ||
| new TestMethodIdentifierProperty( |
There was a problem hiding this comment.
Following generic benchmarks are not shown correctly on VS Test Explorer.
[InProcess]
[GenericTypeArguments(typeof(int))]
[GenericTypeArguments(typeof(int?))]
[GenericTypeArguments(typeof(int[]))]
[GenericTypeArguments(typeof(int?[]))]
[GenericTypeArguments(typeof(int[,]))]
[GenericTypeArguments(typeof(int?[,]))]
public class GenericTypeBenchmarks<T>
{
[Benchmark]
public void Benchmark() { }
}
I though TestMethodIdentifier's property require ECMA-335 compliant type names.
https://learn.microsoft.com/en/dotnet/api/microsoft.testing.platform.extensions.messages.testmethodidentifierproperty
xUnit.net example.
https://github.com/xunit/xunit/blob/rel/4.0.0/src/xunit.v3.common/Extensions/ReflectionExtensions.cs#L171
There was a problem hiding this comment.
It looks like generics are still not displayed as expected.
- Correct NuGet package and namespace in documentation - Add GetBenchmarkUid for stable benchmark identification - Change namespace in BenchmarkCaseIdentityExtensions - Update InternalsVisibleTo for TestingPlatform assembly
Deleted all source, project, and props files from BenchmarkDotNet.TestAdapter.TestingPlatform. This removes all implementation and integration for running benchmarks as tests via Microsoft.Testing.Platform, including test discovery, execution, and result processing logic.
Add BenchmarkDotNet.TestingPlatform.props to enable seamless integration with Microsoft.Testing.Platform. This includes setting required properties for Testing Platform application behavior, ensuring `dotnet test` compatibility on older SDKs, disabling parallel test execution for multi-targeted projects by default, and registering BenchmarkDotNet as a builder hook.
Introduced AsyncWorkQueue in BenchmarkDotNet.TestingPlatform to enable thread-safe, ordered queuing of asynchronous work items. Supports synchronous enqueuing, asynchronous draining, completion signaling, and resource disposal using ConcurrentQueue and SemaphoreSlim.
Introduce a new project to integrate BenchmarkDotNet with Microsoft.Testing.Platform, enabling benchmarks to be discovered and executed as tests. Implements extension identification, test framework, event processing, test node representation, and output logging. Provides builder extensions for easy registration and an MSBuild hook for automatic integration. Updates project configuration for packaging and dependencies.
Updated BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj to reference BenchmarkDotNet.TestingPlatform instead of BenchmarkDotNet.TestAdapter.TestingPlatform. Adjusted both the ProjectReference and Import paths accordingly.
Benchmarks are now grouped by UID to detect collisions. When multiple benchmarks share a UID, `PublishCollisionAsync` reports the issue as a failed test node, allowing other benchmarks to proceed. Only benchmarks with unique UIDs are executed. The refactor introduces a `Match` class, improves cancellation and exception handling, and ensures proper resource cleanup during async operations.
Only whitespace was changed above the GetBenchmarkUid method; no functional or logical modifications were made.
Deleted AsyncWorkQueue.cs, removing the AsyncWorkQueue class and all associated methods for managing and draining asynchronous work items. This eliminates the custom ordered async work queue implementation.
Replaces custom AsyncWorkQueue with ChannelWriter<Func<Task>> for queuing log display tasks. Updates constructor and field types, and switches from Enqueue to TryWrite for task scheduling. This enhances integration with .NET's built-in concurrency primitives.
Switch to System.Threading.Channels for the benchmark event work queue to improve thread safety and prevent deadlocks. Update event processor and logger to use the channel writer, and add a DrainAsync method to process queued work items sequentially until completion.
|
Is it required to keep VSTest based TestAdapter? I think it's better just update the test adapter to use MTP, rather than creating a separate package. It might be cause following limitations by dropping VSTest-based TestAdapter.
Though, It can reduce maintenance cost by dropping VSTest based TestAdapter. |
Standardized benchmark case UID generation by replacing all usages of FullNameProvider.GetBenchmarkUid with BenchmarkCase.GetUniqueId. Removed the obsolete GetBenchmarkUid method. Updated comments for clarity. Also adjusted .slnx to control build for TestingPlatform projects.
|
@filzrev you're right that this is what @timcassell asked for in #2803, so I should settle it before going further. One thing worth separating, though, is that "one package" and "drop VSTest" aren't the same decision. A single package can serve both protocols; that's what MSTest does (EnableMSTestRunner flips the same adapter package into MTP mode), and MS even ships Microsoft.Testing.Extensions.VSTestBridge for frameworks that want one implementation to cover both. BDN wouldn't need the bridge since both implementations already exist here; they'd just ship together, gated by a property. That keeps VS2019 and mixed VSTest/MTP solutions working while still being one package. Merging into BenchmarkDotNet.TestAdapter is mostly repackaging on my side the framework/node/event-processor code moves over intact; what changes is the csproj, the build props, and choosing which entry point to generate. I will be happy to do it. @timcassell @filzrev , which do you want: 1. two packages or 2. one package supporting both protocols? I'll rework to whichever you pick. |
|
"Drop vstest" was implied by my original comment, however if we can have both without too much hassle, I think it's fine. The vstest adapter is considered feature complete afaik, so maintenance should only be a matter of keeping up with any core API changes. I'm not too concerned about 1 vs 2 packages, whatever seems better for user consumption for minimal confusion. If you go with 1 package, the new platform should be the default. |
|
Thanks for the response @timcassell However, one migration detail I want to flag before I do it: the package currently generates a BenchmarkSwitcher.FromAssembly(...).Run(args) entry point (entrypoints/EntryPoint.cs), and in MTP mode Microsoft.Testing.Platform.MSBuild generates its own instead. So on upgrade, dotnet run on an existing benchmark project changes behaviour. I am thinking of accepting it and document it in the changelog since it is simpler and matches "new platform is the default" |
Right, that's why the current vstest adapter disables that (see
We're already shipping lots of breaking changes in 0.16, so it's fine. |
Refactor BenchmarkDotNet.TestingPlatform integration by moving configuration logic from .props to BenchmarkDotNet.TestAdapter.targets. Remove obsolete .props and .csproj files. Update cSpell dictionary to use "testadapter" instead of "testingplatform".
Rewrote and reorganized documentation to focus on the new BenchmarkDotNet.TestAdapter package and its integration with Microsoft.Testing.Platform (MTP) and VSTest. Clarified default behaviors, entry point handling, and configuration steps. Updated code samples and project file snippets. Revised table of contents to reflect the new structure and clarified the relationship between MTP and VSTest. Added notes on IDE support and caveats.
Refactor namespaces from BenchmarkDotNet.TestingPlatform to BenchmarkDotNet.TestAdapter.TestingPlatform throughout the codebase. Update the extension UID and related comments to match the new adapter naming convention and integration targets.
Added explicit imports for BenchmarkDotNet.TestAdapter .props and .targets files in both F# and C# sample projects to ensure adapter build logic is applied. Also imported common.targets. This preserves custom entry points and prevents conversion to Microsoft.Testing.Platform applications.
Switched project reference and build file imports from BenchmarkDotNet.TestingPlatform to BenchmarkDotNet.TestAdapter, including .props and .targets files.
Add NuGet description and set IsTestingPlatformApplication to false to avoid treating the adapter as a test app. Add Microsoft.Testing.Platform.MSBuild as a dependency for downstream projects. Update .props and .targets packaging for cross-platform compatibility. Only generate entry point for VSTest scenarios; clarify comments.
Removed BenchmarkDotNet.TestingPlatform from the solution and deleted its InternalsVisibleTo entry from AssemblyInfo.cs, as it no longer requires access to internal members. No other InternalsVisibleTo changes were made.
|
@timcassell / @filzrev any update on this latest changes for this PR? |
| /// <param name="assemblyPath">The dll or exe of the benchmark project.</param> | ||
| /// <param name="includeJobInName">Whether or not the display name should include the job name.</param> | ||
| /// <returns>The VSTest TestCase.</returns> | ||
| internal static TestCase ToVsTestCase(this BenchmarkCase benchmarkCase, string assemblyPath, bool includeJobInName = false) |
There was a problem hiding this comment.
Is it able to move VSTest specific codes to dedicated directory same as TestingPlatform?
There was a problem hiding this comment.
Yes.
The VSTest code can be moved into a dedicated VSTest/ directory alongside TestingPlatform/. Only BenchmarkEnumerator.cs is shared between the two adapters every other file is VSTest-only and can move.
| // keeps the same identity across processes and across tools. The job is only part of the display name | ||
| // when it actually adds information. | ||
| var uid = benchmarkCase.GetUniqueId(); | ||
| var displayName = $"{fullClassName}.{parametrizedMethodName}" + (includeJobInName ? $" [{jobDisplayInfo}]" : ""); |
There was a problem hiding this comment.
It need to handle Description property of BenchmarkAttribute here to keep compatibility with VSTest-based test adapter.
There was a problem hiding this comment.
Hi @filzrev good catch on the display name. The uid is already covered GetUniqueId hashes Descriptor.DisplayInfo, which includes WorkloadMethodDisplayInfo (the formatted Description), the same input the VSTest adapter's GetTestCaseId uses. The display name isn't: it goes through FullNameProvider.GetMethodName, which is WorkloadMethod.Name + parameters. I'll build the name (and the tree-node path segment) from Descriptor.WorkloadMethodDisplayInfo instead.
There was a problem hiding this comment.
One thing to confirm: TestCase.DisplayName in the VSTest adapter uses GetMethodName too, so it doesn't show Description today either. Do you want me to change both here so they stay in sync, or leave the VSTest side alone in this PR?
There was a problem hiding this comment.
I though TestAdapter is using Description for display name before.
And it's changed to FQDN by PR #2978 to workaround TestExplorer issue.
On MTP mode.
It can separately define DisplayName and TestIdentifierProperty(It must be unique).
So it might be better to use Description property for display name.
| <Project Path="tests/BenchmarkDotNet.IntegrationTests.Static/BenchmarkDotNet.IntegrationTests.Static.csproj" /> | ||
| <Project Path="tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj"> | ||
| <Build Solution="Release|*" Project="false" /> | ||
| </Project> |
There was a problem hiding this comment.
What is the purpose of this setting?
As far as I knows.
TestAdapter can execute benchmark with Debug configuration if benchmark using InProcess toolchain.
There was a problem hiding this comment.
@filzrev This is a leftover from an earlier iteration it shouldn't be there. I've verified the project builds cleanly in Release and that --list-tests and a full run both work against the Release output, and you're right about Debug: BenchmarkEnumerator filters to in-process toolchains when the assembly is unoptimized, and the sample uses InProcessEmitToolchain, so Debug works either way. Removing the element so the project is built in all configurations and is actually covered by CI's Release build.
There was a problem hiding this comment.
Debug: BenchmarkEnumerator filters to in-process toolchains when the assembly is unoptimized,
Although this isn't directly related to this PR,
I'd like to make it possible to disable this behavior via textconfig.json.
Because TestExplorer is expected to be used for debugging and testing.
(It need to Debugger.Break() for csproj-based toolchan though)
| public FastConfig() => AddJob(Job.Dry.WithToolchain(InProcessEmitToolchain.Default)); | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
Is it able to confirm following invalid config test cases?
On my environment, Test discovery is interrupted by exception.
// Specify `abstract` config.
// This benchmark should be silently ignored. (Discovery/Execution should not be stopped by exception)
[Config(typeof(DebugConfig))]
public class Benchmarks_WithAbstractConfigAttribute
{
[Benchmark]
public void Benchmark01()
{
}
}
[Config(typeof(NoPublicConstructorConfig))]
public class Benchmarks_WithInvalidConfig
{
[Benchmark]
public void Benchmark01()
{
}
private class NoPublicConstructorConfig : ManualConfig
{
private NoPublicConstructorConfig() { }
}
}
There was a problem hiding this comment.
Yes confirmed. I reproduced it earlier in this session. Adding both classes to the integration test project and running --list-tests crashes discovery:
Unhandled exception. System.MissingMethodException: Cannot dynamically create an instance of type
'BenchmarkDotNet.Configs.DebugConfig'. Reason: Cannot create an abstract class.
at BenchmarkDotNet.Attributes.ConfigAttribute..ctor(Type type)
at System.Reflection.CustomAttribute.GetCustomAttributes(...)
at BenchmarkDotNet.Helpers.GenericBenchmarksBuilder.BuildGenericsIfNeeded(Type type)
at BenchmarkDotNet.TestAdapter.BenchmarkEnumerator.GetBenchmarksFromAssembly(Assembly assembly)
at BenchmarkDotNet.TestAdapter.TestingPlatform.BenchmarkTestFramework.GetMatchingBenchmarks(...)
The NoPublicConstructorConfig case fails the same way with No parameterless constructor defined for type ....
Two things worth knowing beyond the confirmation:
- It isn't MTP-specific. The throw is in GenericBenchmarksBuilder.BuildGenericsIfNeeded, reached from BenchmarkEnumerator.GetBenchmarksFromAssembly shared with the VSTest adapter. Discovery on master breaks the same way.
- It throws earlier than the obvious fix would catch. It happens while the type list is being built, not in BenchmarkConverter.TypeToBenchmarks, so a try/catch around the conversion wouldn't help. The isolation has to wrap the per-type attribute read.
Replaced the explicit <Build> configuration for BenchmarkDotNet.IntegrationTests.TestingPlatform with a standard <Project> entry to align with the format used for other projects in the solution.
| @@ -20,6 +28,13 @@ | |||
| <PackageReference Include="Microsoft.TestPlatform.TranslationLayer" Version="17.8.0" /> | |||
There was a problem hiding this comment.
Following comment is leaved as not and it's not be handled on this PR
These VSTest specific packages should be referenced when using VSTest mode.
If's better remove these PackageReference on MTP mode.
Expected behavior
- By default, It should use MTP mode and VSTest related package references are removed by custom MSBuild targets.
- If VSTest mode. It should raise error that instruct explicitly specify VSTestMode on PackageReference.
Replaced default introduction with a personalized message identifying as GitHub Copilot and offering software development assistance.
Refactored the logic for generating method display names in BenchmarkDotNet. Improved separation of concerns by extracting display name generation to a dedicated provider. Updated the display name formatting to include job information conditionally. Enhanced maintainability and clarity in the test adapter's method identification process.
Introduced DescribedProbe to test benchmarks with and without custom descriptions. Includes FastConfig for quick in-process execution and a configurable Size parameter.
3 participants
#2803
@timcassell
Introduce BenchmarkTestFramework to integrate BenchmarkDotNet with Microsoft.Testing.Platform, enabling benchmark discovery and execution. Implements session management, filtering, event processing, and output routing. Handles test node updates and cancellation, with support for experimental platform features.
Added BenchmarkDotNet.TestAdapter.TestingPlatform and BenchmarkDotNet.IntegrationTests.TestingPlatform projects to the solution. Updated MonoBenchmarks and SharedDiagnosers integration test projects to only build for the solution in Debug configuration.