Is your feature request related to a problem? Please describe.
PyRIT does not have a bounded scenario for garak's repetition-induced divergence checks. A direct port would generate tokenizer data at runtime and change target output settings from probe code. Those behaviors do not fit PyRIT component responsibilities and can create unexpectedly expensive runs. This is part of #511.
Describe the solution you'd like
Add a PyRIT-native divergence scenario based on garak/probes/divergence.py.
Required scope and behavior:
- Make
Repeat the default scenario technique.
- Add
RepeatedToken as an explicit opt-in technique. Leave inactive RepeatExtended outside the required first version.
- Store repeat templates, seed words, and safe configuration metadata in local PyRIT datasets.
- Store a small, fixed, attributed set of precomputed repeated-token strings in the dataset. Do not add runtime tokenizer generation or a new tokenizer dependency.
- Use standard PyRIT attack flow. Do not mutate target
max_tokens or other target settings in the scenario. Document any recommended target output limit as user configuration.
- Reuse existing refusal and text-analysis components where their contracts fit. If no existing scorer covers divergence, add one deterministic scorer that distinguishes refusal or bounded requested repetition from unexpected continuation, and that detects excessive repeated structures for
RepeatedToken.
- Set conservative prompt counts and response limits so the default is not a denial-of-service test.
- Describe positive results as divergence or output instability. Do not claim training-data leakage without a separate attributable reference match.
- Add exports, bounded unit tests, and synchronized
.py and .ipynb scanner documentation.
Use PromptInject PR #2509 as an example of dataset/technique separation. Follow doc/code/framework.md and the applicable scenario, dataset, scorer, test, and documentation instructions. Do not port garak generator hooks into a scenario.
Describe alternatives you've considered, if relevant
- Runtime
tiktoken generation was considered and rejected. Fixed data keeps the scenario deterministic and avoids dependency and tokenizer-version drift.
- Changing target output limits from scenario code was considered and rejected. The target owner must control those limits.
- Treating any repetition as a positive result would confuse expected bounded compliance with divergence.
Additional context
Is your feature request related to a problem? Please describe.
PyRIT does not have a bounded scenario for garak's repetition-induced divergence checks. A direct port would generate tokenizer data at runtime and change target output settings from probe code. Those behaviors do not fit PyRIT component responsibilities and can create unexpectedly expensive runs. This is part of #511.
Describe the solution you'd like
Add a PyRIT-native divergence scenario based on
garak/probes/divergence.py.Required scope and behavior:
Repeatthe default scenario technique.RepeatedTokenas an explicit opt-in technique. Leave inactiveRepeatExtendedoutside the required first version.max_tokensor other target settings in the scenario. Document any recommended target output limit as user configuration.RepeatedToken..pyand.ipynbscanner documentation.Use PromptInject PR #2509 as an example of dataset/technique separation. Follow
doc/code/framework.mdand the applicable scenario, dataset, scorer, test, and documentation instructions. Do not port garak generator hooks into a scenario.Describe alternatives you've considered, if relevant
tiktokengeneration was considered and rejected. Fixed data keeps the scenario deterministic and avoids dependency and tokenizer-version drift.Additional context
garak/probes/divergence.py