Scenarios

Multi-step workflow testing with scenario builders and runners.

`Scenario`

Module: giskard.checks.core.scenario

Create a scenario, add interactions and checks, then call .run(). Each method returns the same scenario. Use .extend() to add existing specs or checks, or pass the scenario to Suite.append().

Scenario() → Scenario

Create a new scenario.

name str Default: Unnamed Scenario

Scenario name for identification. Pass the first argument positionally as in Scenario(“my_name”).

trace_type type[TraceType] | None Default: None

Optional custom trace type for advanced use cases.

multiple_runs int Default: 1

Default cap on full scenario executions when run() is called without multiple_runs=…. Must be ≥ 1.

tags list[str] Default: []

Labels for grouping and Hub upload. Strings in Key:Value form; tags without : are bare labels.

.interact() → self

Add an interaction to the scenario. Returns self for chaining.

inputs value | Callable Required

Static value or callable (trace) -> value.

outputs value | Callable

Static value, callable (inputs) -> value, or (trace, inputs) -> value. Optional when the scenario / suite has a target.

metadata dict | None

Optional metadata dictionary.

.check() → self

Add a validation check to the scenario. Returns self for chaining.

check Check Required

A Check instance to validate the trace.

.add_interaction() → self

Add a pre-constructed InteractionSpec object.

interaction InteractionSpec Required

The interaction spec to add.

.extend() → self

Append one or more interaction specs and/or checks. Returns self for chaining.

*components InteractionSpec | Check Required

Components to append in order.

.with_tags() → self

Set scenario tags for grouping and Hub upload. Replaces any tags already set. Returns self for chaining.

tags list[str] Required

Flat strings in Key:Value format. Tags without : are bare labels.

.run() → ScenarioResult

Execute the scenario against the SUT and return results.

target Callable | None Default: None

Override the scenario’s default target system-under-test for this run.

return_exception bool Default: False

If True, return results even when exceptions occur instead of raising.

multiple_runs int | None Default: None

When set, overrides the scenario’s multiple_runs field: maximum full scenario executions (fresh trace each time). Each run must pass for the next to run; stops on the first FAIL, ERROR, or SKIP. Not a retry-until-success loop.

Multi-step example

Each step can add interactions and then validate the trace:

from giskard.checks import Scenario, FnCheck, Equals

result = await (
    Scenario("two_step_example")
    .with_tags(["example:multi-step"])
    .interact(
        inputs="Find item 42",
        outputs={"answer": "Which collection?", "found": False},
    )
    .check(
        FnCheck(
            fn=lambda trace: trace.last is not None
            and "?" in trace.last.outputs["answer"],
            name="asks_follow_up",
        )
    )
    .interact(
        inputs="In the archive",
        outputs={"answer": "Item found", "found": True},
    )
    .check(Equals(expected_value=True, target_key="trace.last.outputs.found"))
    .run()
)

Dynamic interactions

Pass a callable as outputs to run the agent instead of replaying a recorded reply:

from giskard.checks import Scenario


def bank_support_agent(inputs: str) -> str:
    """Stand-in for the agent under test. Replace with your own call."""
    if "transfer" in inputs.lower():
        return "A SEPA transfer arrives within one working day and is free."
    return "Could you tell me a bit more about what you need?"


scenario = Scenario("transfer_timing").interact(
    inputs="How long does a SEPA transfer take?", outputs=bank_support_agent
)

Context-aware interactions

Pass a callable as inputs when the next message depends on what the agent said:

scenario = (
    Scenario("bank_agent_follow_up")
    .interact(
        inputs="How long does a SEPA transfer take?",
        outputs=bank_support_agent,
    )
    .interact(
        inputs=lambda trace: f"You said: {trace.last.outputs} Is that the same at a weekend?",
        outputs=bank_support_agent,
    )
)

`ScenarioResult`

Module: giskard.checks.core.result

Result of scenario execution with trace and check results.

ScenarioResult

scenario_name str

Name of the scenario that produced this result.

status ScenarioStatus

Overall status (PASS/FAIL/ERROR/SKIP).

steps list[TestCaseResult]

Results for each step (interactions in that step, then checks).

final_trace Trace

Complete trace of all interactions.

passed bool

True when the aggregated status is PASS: no failures or errors, and not all steps skipped (or empty).

failed bool

True when at least one step failed and none errored.

errored bool

True when at least one step errored.

skipped bool

True when all steps were skipped.

duration_ms int

Total execution time in milliseconds.

multiple_runs int

Configured cap on full scenario executions for this invocation (from the scenario or the run(multiple_runs=...) override).

runs_executed int

How many full scenario executions ran before stopping (at most multiple_runs).

tags list[str]

Snapshot of the scenario’s tags at run time. Used by SuiteResult.group_by() and Hub upload.

.print_report() → None

Print the result to the terminal with Rich, including the trace and every check verdict. Inherited from BaseResult, so CheckResult, TestCaseResult, and SuiteResult all have it.

console Console | None Default: None

Rich Console to print to. Defaults to a new one writing to stdout.

Using the scenario built above:

result = await scenario.run()
result.print_report()

if result.passed:
    print("All checks passed!")

print(f"Total interactions: {len(result.final_trace.interactions)}")

for i, check_result in enumerate(
    r for step in result.steps for r in step.results
):
    print(f"Check {i}: {check_result.status}")

`ScenarioStatus`

Module: giskard.checks.core.result (also exported from giskard.checks)

Outcome categories for a scenario execution. Derived from the scenario’s steps: ERROR if any step errored, else FAIL if any step failed, else SKIP if every step was skipped, else PASS. An empty scenario is PASS.

Status	Value	Description
`PASS`	`"pass"`	No failures or errors; not all steps skipped (or empty)
`FAIL`	`"fail"`	At least one step failed and none errored
`ERROR`	`"error"`	At least one step errored
`SKIP`	`"skip"`	All steps were skipped

from giskard.checks import ScenarioStatus

if result.status == ScenarioStatus.PASS:
    print("Success!")

`Suite`

Module: giskard.checks.scenarios.suite

Group multiple scenarios and run them together.

Suite() → Suite

name str Required

Suite identifier.

target Callable

Optional suite-level target SUT.

.append() → Suite

Add a scenario to the suite.

scenario Scenario Required

The scenario to add.

.run() → SuiteResult

Run every scenario in the suite.

target Callable

Override target for this run. Overrides both the suite-level target and any scenario-level target.

return_exception bool Default: False

Return results on exceptions.

parallel bool Default: False

Run scenarios concurrently while preserving result order. The scan helpers pass parallel=True; Suite.run on its own is serial.

max_concurrency int | None Default: None

Cap on concurrent scenarios when parallel=True. None starts them all at once, so your provider’s rate limit becomes the real cap.

verbose bool Default: True

Show a progress bar naming the running scenario. Set to False in CI.

Set the target once on the suite and leave outputs off each interaction:

from giskard.checks import Scenario, StringMatching, Suite


def target(inputs: str) -> str:
    return f"Received: {inputs}"


suite = Suite(name="examples", target=target)
suite.append(
    Scenario("contains_received")
    .with_tags(["example:greeting"])
    .interact(
        inputs="Hello"
    ).check(
        StringMatching(
            keyword="Received", target_key="trace.last.outputs"
        )
    )
)
suite.append(
    Scenario("another_input").with_tags(["example:greeting"]).interact(inputs="Goodbye")
)

result = await suite.run()

rate = result.pass_rate
print("no scenarios evaluated" if rate is None else f"{rate:.0%}")

pass_rate is the fraction of non-skipped scenarios that passed.

`SuiteResult`

Module: giskard.checks.core.result

Aggregate result from suite execution.

SuiteResult

results list[ScenarioResult]

Scenario results in order.

pass_rate float | None

Fraction of non-skipped scenarios that passed. None when nothing was evaluated: an empty suite, or one where every scenario was skipped.

duration_ms int

Total execution time in milliseconds.

passed_count int

Number of passed scenarios.

failed_count int

Number of failed scenarios.

errored_count int

Number of scenarios that errored.

skipped_count int

Number of scenarios that were skipped.

suite Suite | None Default: None

The Suite that produced this result. Excluded from serialization (None after a serialize/deserialize round-trip).

recommendation str | None Default: None

Optional Markdown-friendly guidance attached by scan or suite producers. Rendered in print_report() when set.

.group_by() → GroupedSuiteResult

Group results by a tag key. A scenario may appear in multiple buckets if it carries several tags with the same key; totals across buckets can exceed the number of scenarios. Scenarios with no matching tag go into the None bucket.

key str Required

Tag key to group by (the part before :, e.g. “threat-type”).

.print_report() → None

Print the suite report. Overrides BaseResult.print_report() with an optional grouped pass-rate table.

console Console | None Default: None

Rich Console to print to. Defaults to a new one writing to stdout.

group_by str | None Default: None

Tag key to group by. When set, appends a per-group pass-rate table after the standard report.

.to_hub_format() → dict[str, Any]

Convert the suite result into a JSON-serializable Giskard Hub payload. Pass the dict to giskard_hub.HubClient.evaluations.upload().

.to_junit_xml() → str

Export the suite result as a JUnit XML string. Optionally write to a file.

path str | Path | None Default: None

File path to write the XML to. Returns the XML string regardless.

grouped = result.group_by("example")
result.print_report(group_by="example")
payload = result.to_hub_format()

`GroupStats`

Module: giskard.checks.core.result (also exported from giskard.checks)

Pass/fail counts for one tag-value bucket. Mirrors Hub’s Metric shape.

GroupStats

name str | None

Tag value for this bucket, or None for untagged scenarios.

passed int

Scenarios in this bucket that passed.

failed int

Scenarios in this bucket that failed.

errored int

Scenarios in this bucket that errored.

skipped int Default: 0

Scenarios in this bucket that were skipped.

total int

passed + failed + errored + skipped.

non_skipped int

Scenarios counted toward pass rate (total - skipped).

pass_rate float | None

Fraction passed out of non_skipped. None when non_skipped == 0.

`GroupedSuiteResult`

Module: giskard.checks.core.result (also exported from giskard.checks)

SuiteResult grouped by a tag key, with per-group stats. Returned by SuiteResult.group_by(). Rich rendering prints the suite report, then a pass-rate table titled with that key. Untagged buckets display as (untagged); an empty tag value displays as true.

GroupedSuiteResult

suite_result SuiteResult

The original ungrouped suite result.

key str

Tag key used for grouping.

groups dict[str | None, GroupStats]

Per-value stats. The None key holds scenarios with no matching tag.

`ScenarioRunner`

Module: giskard.checks.scenarios.runner

Low-level runner for executing scenarios. Most users should use Scenario(...).run() instead.

.run() → ScenarioResult

scenario Scenario Required

The scenario to execute.

target Callable | None Default: None

Override the scenario’s target SUT.

return_exception bool Default: False

Return results on exceptions.

multiple_runs int | None Default: None

Optional override of the scenario’s multiple_runs (same semantics as Scenario.run(multiple_runs=...)).

get_runner() → ScenarioRunner

Get the default process-wide singleton runner instance.

Scenarios

Scenario

Multi-step example

Dynamic interactions

Context-aware interactions

ScenarioResult

ScenarioStatus

Suite

SuiteResult

GroupStats

GroupedSuiteResult

ScenarioRunner

See also

`Scenario`

`ScenarioResult`

`ScenarioStatus`

`Suite`

`SuiteResult`

`GroupStats`

`GroupedSuiteResult`

`ScenarioRunner`