Reorganize command line interface options/parameters #3182 - #3216
Reorganize command line interface options/parameters #3182#3216sheddy123 wants to merge 14 commits into
Conversation
Added documentation for the new `--profile` console argument, detailing how to select benchmark jobs by Id, use glob patterns, and distinguish it from `--profiler`. Included code and command-line examples, and updated the options summary.
Added tests to BenchmarkSwitcherTest.cs covering --profile usage: matching job IDs, glob patterns, attribute-defined jobs, error messages for unmatched profiles, and combined profile/filter scenarios. Introduced WithTwoNamedJobAttributes for attribute-based job selection tests.
Introduced JobIdFilterTests to validate filtering benchmarks by job ID, including exact, case-insensitive, glob patterns, and attribute-defined jobs. Tests also verify recording of observed and matched job IDs. Added helper benchmark classes for test scenarios.
Added tests to ensure correct handling of --profile and --filter arguments in BenchmarkDotNet config parsing. Verified that JobIdFilter is only added when a profile is specified, profiles are case-insensitive, and profile filters are not unioned with other filters. Improves test coverage for command-line argument parsing.
Introduced JobIdFilter in BenchmarkDotNet.Filters to enable filtering benchmarks using job IDs with glob pattern support. The filter tracks observed and matched job IDs, exposes them for reporting, and applies filtering logic via regular expressions in the Predicate method.
Introduced a Profiles command-line option to CommandLineOptions, enabling users to run only jobs with matching Id(s) using glob patterns. Updated help text to clarify its distinction from the Profiler option and added a usage example for --profile.
Ensure JobIdFilter is added when profiles are specified, so job selection by profile is always combined with other filters. Added comments to clarify filter handling logic.
Added TryPrintWrongProfileInfo to clarify when --profile returns no benchmarks, providing targeted error messages and listing available profiles. Integrated this logic into the filtering process and limited displayed profiles to 40. Added necessary using directive for BenchmarkDotNet.Filters.
|
This PR appears to provide a filtering feature based on JobId. It's a bit different feature that I've commented at #3182. I personally thought it's better to provide By using |
Replaced all references to the --profile argument with --filterJobId for filtering benchmark jobs by Id. Updated examples, descriptions, and usage instructions to reflect this change and clarified usage details.
Replaces the --profile option and Profiles property with --filterJobId and FilterJobIds in CommandLineOptions. Updates all references, help text, examples, and filter logic to use the new naming, clarifying that the filter applies to job IDs rather than profiles.
Replaces all usage of the "--profile" argument with "--filterJobId" in test methods, assertions, and log messages. Updates method names and expectations to use "JobId" terminology for consistency across BenchmarkSwitcherTest.cs and ConfigParserTests.cs.
Refactor BenchmarkSwitcher to use "job Id"/"--filterJobId" instead of "profile"/"--profile" throughout variable names, methods, comments, and user messages. Rename MaxDisplayedProfiles to MaxDisplayedJobIds. Update BenchmarkDotNet.slnx to exclude MonoBenchmarks and SharedDiagnosers integration tests from Release builds by default.
|
@filzrev does this suffice |
|
It looks good to me about the PR content that adding If there are no other issues, please go ahead and merge it. Note: Original issue's intent is to reorganize command line arguments and reduce options. Example
|
|
Hi @filzrev thanks for the review Adding the advantage of adding --filterJobId option is makes the CLI semantics unambiguous, enables distinct error messages, and reflects the real distinction between "which benchmarks?" and "in which jobs?" |
|
I'm hesitant to merge this. Adding a bunch of different --filter* arguments is smelly. I also am not happy using job ID (I mentioned a while back that it's a bad API). We should probably do something more like job categories, and expand the --filter query instead of new filter arguments like @filzrev suggested. |
|
@timcassell Thanks for the update. I was wondering if this is what you meant:
|
3 participants
Issue #3182's
@filzrev based on your suggestions, "I have specified a benchmark profile key from the command line".
A profile is now a job Id, matched with globs, case-insensitively:
dotnet run -c Release -- --filter 'JsonSerialization' --profile debug
dotnet run -c Release -- --filter '*' --profile net8 net9
Ids come from Job.Default.WithId("net8") in code or [SimpleJob(RuntimeMoniker.Net80, id: "net8")].
Job.Id already existed but was only ever used for display, folder names, and ordering, never selection.