Testing Utilities

Testing utilities, test runners, and debugging helpers.

---

TestCase

Module: giskard.checks.core.testcase

Bundle a trace with a set of checks to execute. Useful for testing against fixed interaction sequences or replaying recorded conversations.

Note

For most use cases, use the Scenario() fluent API instead. TestCase is for advanced use cases where you need direct control over trace construction.

TestCase

name str | None

Optional label for the test case.

trace Trace Required

The trace containing interactions to test against.

checks Sequence[Check] Required

Sequence of checks to run against the trace.

.run() → TestCaseResult

Execute all checks against the trace.

return_exception bool Default: False

If True, return results even when exceptions occur instead of raising.

.assert_passed() → None

Run the test case and assert that all checks passed. Raises AssertionError with formatted failure messages if any check fails.

A conversation you captured in production, replayed against checks without calling the agent again:

from giskard.checks import TestCase, Trace, Interaction, Equals

trace = Trace(
    interactions=[
        Interaction(
            inputs="What is the monthly fee on my current account?",
            outputs={"intent": "fee_enquiry", "fee_eur": 3.00},
        ),
        Interaction(
            inputs="And to replace a lost card?",
            outputs={"intent": "fee_enquiry", "fee_eur": 0.00},
        ),
    ]
)

test_case = TestCase(
    name="fee_enquiry_replay",
    trace=trace,
    checks=[
        Equals(
            expected_value=3.00,
            target_key="trace.interactions[0].outputs.fee_eur",
        ),
        Equals(
            expected_value=0.00,
            target_key="trace.interactions[1].outputs.fee_eur",
        ),
    ],
)

result = await test_case.run()

Integration with pytest

assert_passed() raises AssertionError with the check messages, so a failing check reads like any other pytest failure:

import pytest

from giskard.checks import FnCheck, Interaction, TestCase, Trace


@pytest.mark.asyncio
async def test_vague_question_gets_a_follow_up():
    trace = Trace(
        interactions=[
            Interaction(
                inputs="Something is wrong with my account.",
                outputs="Sorry to hear that. Is it a payment, a card, or the balance?",
            )
        ]
    )
    test_case = TestCase(
        name="asks_a_follow_up",
        trace=trace,
        checks=[
            FnCheck(
                fn=lambda t: t.last is not None and "?" in t.last.outputs,
                name="asks_a_follow_up",
            ),
        ],
    )
    await test_case.assert_passed()

---

TestCaseResult

Module: giskard.checks.core.result

Result of test case execution with check outcomes.

TestCaseResult

status TestCaseStatus

Overall test case status (PASS/FAIL/ERROR/SKIP).

results list[CheckResult]

Results from all checks.

passed bool

True when status is PASS: no failures or errors, and not all checks skipped (or empty).

failed bool

True if any check failed.

errored bool

True if any check errored.

skipped bool

True if all checks were skipped.

duration_ms int

Execution time in milliseconds.

last_interaction_index int | None Default: None

0-based index, in the scenario’s final trace, of the last interaction this step added before its checks ran. None when the step added no interactions (for example, skipped). Hub upload uses this to attribute check results to a specific interaction.

error TestCaseError | None Default: None

Execution error that prevented the test case from running normally, such as an input-generation failure.

.assert_passed() → None

Raise AssertionError with formatted failure messages if any check did not pass. Useful in pytest.

result = await test_case.run()

if result.passed:
    print("All checks passed!")
else:
    for check_result in result.results:
        if check_result.failed:
            print(f"Failed: {check_result.message}")

# Assert (raises if failed)
result.assert_passed()

---

TestCaseStatus

Module: giskard.checks.core.result (also exported from giskard.checks)

Outcome categories for a test case execution. Derived from the case’s check results (and error, if set): ERROR if error is set or any check errored, else FAIL if any check failed, else SKIP if every check was skipped, else PASS. An empty test case is PASS.

Status	Value	Description
PASS	"pass"	No failures or errors; not all checks skipped (or empty)
FAIL	"fail"	At least one check failed and none errored
ERROR	"error"	At least one check errored, or error is set
SKIP	"skip"	All checks were skipped

from giskard.checks import TestCaseStatus

if result.status == TestCaseStatus.PASS:
    print("Success!")

---

TestCaseError

Module: giskard.checks.core.result (also exported from giskard.checks)

Captures why a test case failed to execute.

TestCaseError

message str

Error message.

exception_type str

Exception class name (for example "ValueError").

traceback str | None Default: None

Optional traceback string.

phase str | None Default: None

Optional execution phase during which the error occurred.

.summary() → str

One-line description: the exception type, optional phase, then the message.

---

TestCaseRunner

Module: giskard.checks.testing.runner

Low-level runner for executing test cases. Most users should use test_case.run() instead.

.run() → TestCaseResult

Execute a test case’s checks against its trace.

test_case TestCase Required

The test case to execute.

return_exception bool Default: False

Return results on exceptions.

get_runner() → TestCaseRunner

Get the default process-wide singleton runner instance.

---

WithSpy

Module: giskard.checks.testing.spy

Replace a function with a MagicMock while the wrapped interaction spec runs, then record what it was called with. Use it to assert that your agent passed the right arguments to an internal call.

WithSpy

interaction_generator InteractionSpec Required

The interaction spec to spy on.

target str Required

Python import path of the function to patch, exactly as you would pass it to unittest.mock.patch (for example "myapp.db.fetch_orders"). This is not a JSONPath — a trace expression such as "trace.last.outputs" raises.

After each interaction, the mock’s call history is written to Interaction.metadata under the target string, then the mock is reset. The recorded keys are call_count, call_args, call_args_list, and mock_calls.

Assert that the target called a dependency:

from giskard.checks import Scenario, Interact, WithSpy

from myapp.agent import answer

interaction_spec = Interact(
    inputs="Find item 42",
    outputs=answer,
)

spied_spec = WithSpy(
    interaction_generator=interaction_spec,
    target="myapp.db.fetch_item",  # Python import path, same as mock.patch
)

result = await Scenario("dependency_call").add_interaction(spied_spec).run()

# Spy data is keyed by the same import path
spy_data = result.final_trace.last.metadata.get("myapp.db.fetch_item")
print(spy_data["call_count"], spy_data["call_args"])

Tip

Use WithSpy when debugging complex interaction generation logic, understanding multi-turn interaction flow, or investigating unexpected behavior in scenarios.

---

Usage patterns

Replaying recorded conversations

Replay a recorded transcript:

from giskard.checks import TestCase, Trace, Interaction, FnCheck

recorded = [
    Interaction(inputs="Find item 42", outputs="Item found."),
    Interaction(inputs="Who created it?", outputs="Alex created it."),
]

trace = Trace(interactions=recorded)

test_case = TestCase(
    name="recorded_replay",
    trace=trace,
    checks=[
        FnCheck(
            fn=lambda t: "found" in t.interactions[0].outputs.lower(),
            name="finds_item",
        ),
        FnCheck(
            fn=lambda t: "Alex" in t.interactions[1].outputs,
            name="returns_creator",
        ),
    ],
)

await test_case.assert_passed()

Replay does not call the agent.

Batch testing

Run independent test cases concurrently:

import asyncio

test_cases = [
    TestCase(
        name=f"replay_{i}",
        trace=Trace(interactions=[interaction]),
        checks=[
            FnCheck(
                fn=lambda t: t.last is not None and bool(t.last.outputs),
                name="answered",
            )
        ],
    )
    for i, interaction in enumerate(recorded)
]

results = await asyncio.gather(*[tc.run() for tc in test_cases])

passed = sum(1 for r in results if r.passed)
print(f"Passed: {passed}/{len(results)}")

Parameterized tests

One check, many recorded exchanges:

import pytest

from giskard.checks import Interaction, StringMatching, TestCase, Trace


def target(inputs: str) -> str:
    return f"Received: {inputs}"


test_data = [
    ("Hello", "Received"),
    ("Goodbye", "Received"),
]


@pytest.mark.asyncio
@pytest.mark.parametrize("question,required_keyword", test_data)
async def test_answers_include_keyword(question, required_keyword):
    trace = Trace(
        interactions=[
            Interaction(inputs=question, outputs=target(question))
        ]
    )
    test_case = TestCase(
        name=f"keyword_{required_keyword}",
        trace=trace,
        checks=[
            StringMatching(
                keyword=required_keyword,
                target_key="trace.last.outputs",
                case_sensitive=False,
            )
        ],
    )
    await test_case.assert_passed()

---

See also

• Core API — Trace, Interaction, and Check details
• Scenarios — Building multi-step test workflows
• Built-in Checks — Ready-to-use validation checks