Skip to content

FEAT: Add Garak exploitation scenario #2534

Description

Is your feature request related to a problem? Please describe.

PyRIT does not package garak's active lightweight server-side template injection and SQL payload-echo checks as a scenario. Users can call some related scorers directly, but there is no bounded campaign that owns the payload datasets and applies the correct success contract. This is part of #511.

Describe the solution you'd like

Add a PyRIT-native exploitation scenario for the active JinjaTemplatePythonInjection and SQLInjectionEcho variants in exploitation.py.

Implementation requirements:

  • Store the SQL and Python/Jinja payloads in local PyRIT datasets. Store reusable wrapping or echo templates in dataset-owned prompt templates instead of hard-coding the corpus in the scenario.
  • Preserve the payload, exploit family, expected emitted text, and source provenance in seed metadata.
  • Use a small dataset configuration to construct attack groups and use the standard PromptSendingAttack for the single-turn requests.
  • Match the upstream primary contract: a result is positive when the target emits the expected payload material. Compose a payload-specific SubStringScorer for this exact check.
  • Reuse SQLInjectionOutputScorer and SSTIOutputScorer only as optional extended checks where their existing contracts fit. Do not broaden them only to absorb this corpus, and do not add a duplicate garak detector by default.
  • Report that the model emitted exploit material. Do not claim that a downstream SQL database or template engine was exploited.
  • Use deterministic, bounded defaults. Add scenario exports, focused unit tests, and synchronized .py and .ipynb scanner documentation.

Follow the separation in PromptInject PR #2509 and doc/code/framework.md: datasets own payloads and templates, techniques describe application, the scenario assembles attacks, attacks execute, and scorers evaluate. Do not port garak's probe, hook, or detector runtime.

Describe alternatives you've considered, if relevant

  • SQLInjectionSystem is not required in the first version. Its contract needs inspection of a real downstream database and is not a self-contained target-response check.
  • Using only the existing regex scorers was considered. Their current patterns do not implement every payload-specific echo check in the upstream corpus.
  • Porting garak's probe and detector classes would duplicate PyRIT execution and scoring responsibilities.

Additional context

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions