[FEAT]: Replace trial cloning with configurable trial populations

[Behnam (behnam-o)]

Summary

Replace pytest item cloning for @pytest.mark.trial with an explicit trial_config fixture. Tests control population execution while RAMPART supplies:

• The marker-declared population size and threshold
• A --rampart-trials N CLI override for population size
• Collection-time validation of trial declarations
• An error when a trial-marked test does not consume trial_config

This avoids clone-specific pytest and xdist behavior while preserving population-level result reporting.

Changes

• Add the immutable TrialConfig public type
• Add the trial_config fixture
• Add the --rampart-trials N option
• Remove collection-time test cloning
• Validate trial markers during collection
• Update xdist tests for one-item, multiple-result populations
• Update trial documentation and examples

Example

from rampart import Probes, execute_trials_async

@pytest.mark.trial(n=10, threshold=0.8)
async def test_injection_resistance(adapter, trial_config):
    population = await execute_trials_async(
        execution_factory=lambda: Probes.behavior(...),
        adapter=adapter,
        n=trial_config.n,
        threshold=trial_config.threshold,
    )
    assert population, population.summary

This PR now includes the prerequisite execution-population work from #121 through the latest main merge.

[Copilot]

Pull request overview

This PR updates RAMPART’s pytest @pytest.mark.trial semantics to stop cloning pytest items and instead supply trial population configuration through a dedicated trial_config fixture (with an optional --rampart-trials N CLI override). This avoids clone/xdist scheduling edge-cases while keeping population-style reporting.

Changes:

• Introduces an immutable TrialConfig type and trial_config fixture, plus --rampart-trials N to override population depth (n) without changing the declared threshold.
• Removes collection-time trial cloning and replaces it with collection-time validation (including enforcing that trial-marked tests consume trial_config).
• Updates unit/integration/xdist tests and refreshes documentation/examples to reflect population execution being controlled by the test.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/unit/pytest_plugin/test_xdist_aggregation.py	Updates xdist aggregation tests to reflect “single pytest item that records multiple Results” populations.
tests/unit/pytest_plugin/test_trial.py	Adds unit tests for trial marker parsing/validation and CLI override resolution.
tests/unit/pytest_plugin/test_trial_integration.py	Adds subprocess/integration tests verifying fixture behavior and failure modes.
tests/unit/pytest_plugin/test_plugin.py	Removes cloning-related unit tests that no longer apply.
rampart/reporting/sink.py	Updates population_summary docstring to reflect multi-Result-per-test behavior.
rampart/pytest_plugin/plugin.py	Adds --rampart-trials, adds trial_config fixture, removes cloning logic, and enforces trial marker validation + fixture consumption.
rampart/pytest_plugin/_trial.py	New module implementing TrialConfig, CLI parsing helper, and marker/override resolution.
rampart/pytest_plugin/_session.py	Updates docstrings to clarify trial-spec paths are legacy/compatibility only.
rampart/pytest_plugin/init.py	Exposes TrialConfig on the public pytest plugin API.
pyproject.toml	Updates marker description to the new population/fixture semantics.
docs/usage/xdist.md	Updates xdist guidance to reflect that @trial no longer affects scheduling; adds --rampart-trials guidance.
docs/usage/pytest-integration.md	Updates marker semantics, adds trial_config fixture docs, updates examples and sample output.
docs/usage/configuration.md	Documents --rampart-trials N alongside other pytest options.
docs/usage/ci-integration.md	Updates CI guidance/examples to use trial_config and CLI override.
docs/usage/authoring-tests.md	Updates authoring guidance/examples to show trial_config usage in class-based tests.
docs/probes/behavioral.md	Updates probe example to execute a population via trial_config.
docs/glossary.md	Updates “Trial” definition to match population + trial_config semantics.
docs/getting-started/quickstart.md	Updates quickstart example/output to use trial_config and population execution pattern.
docs/getting-started/installation.md	Updates expected marker help output text.
docs/concepts/overview.md	Updates overview wording for trial marker semantics.
docs/attacks/xpia.md	Updates attack example to execute a population via trial_config.
docs/api/pytest-plugin.md	Adds _trial module docs entry (but currently misreferences TrialGroupResult).

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Spencer Schoenberg (spencrr)]

I think this only enforces that the caller imports the fixture but not necessarily that they forward the args. Not that we necessarily need to strictly enforce this, but execute_trials_async could instead (1) take a TrialConfig (although this would require that every trial call site be fully configurable by CLI flag with required plumbing) or (2) refactor execute_trials_async as a fixture that depends on the trial_config fixture - what do you think?