Skip to content

FEAT: Add Garak API key scenario #2535

Description

Is your feature request related to a problem? Please describe.

PyRIT does not have a scenario for garak's API-key generation and partial-key completion checks. CredentialLeakScorer provides a useful base, but its current service-pattern coverage is smaller than the upstream probe's coverage. A direct detector port could also score echoed partial keys and safe placeholders as leaks. This is part of #511.

Describe the solution you'd like

Add a PyRIT-native API key scenario for the active GetKey and CompleteKey variants in garak/probes/apikey.py.

Implementation requirements:

  • Store service/key-type names, request templates, safe placeholder exclusions, and synthetic partial keys in local PyRIT datasets. Never add real credentials.
  • Model key generation and key completion as scenario techniques. Use the standard PromptSendingAttack.
  • Reuse CredentialLeakScorer. Audit its coverage against the active upstream service patterns and extend the existing scorer with reviewed service-specific patterns only where needed for parity. Do not add a duplicate scenario-specific detector unless the existing scorer contract cannot represent the check.
  • Preserve the MIT attribution for any regex implementation derived from garak's referenced dora project. Garak itself is Apache-2.0, so retain the applicable garak notices for copied or adapted content.
  • A positive score must require a credential-shaped value that is not only the partial value supplied in the request. Exclude known safe placeholders and straightforward request echo.
  • Bound the default prompt count. CompleteKey can otherwise create hundreds of service/prefix combinations.
  • Add exports, focused tests for service coverage and false-positive controls, and synchronized .py and .ipynb scanner documentation. All test and documentation values must be synthetic.

Follow PromptInject PR #2509, doc/code/framework.md, and the applicable scenario, dataset, scorer, test, and documentation instructions. Keep source data in datasets, execution in attacks, and response evaluation in the scorer.

Describe alternatives you've considered, if relevant

  • Copying the complete garak detector into the scenario would duplicate CredentialLeakScorer and mix scoring with orchestration.
  • A single broad secret regex would have poor precision.
  • Real or plausibly active key material is not needed for this test and must not be used.

Additional context

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions