Testing Utilities

Testing utilities, test runners, and debugging helpers.

`TestCase`

Module: giskard.checks.core.testcase

Bundle a trace with a set of checks to execute. Useful for testing against fixed interaction sequences or replaying recorded conversations.

TestCase

name str | None

Optional label for the test case.

trace Trace Required

The trace containing interactions to test against.

checks Sequence[Check] Required

Sequence of checks to run against the trace.

.run() → TestCaseResult

Execute all checks against the trace.

return_exception bool Default: False

If True, return results even when exceptions occur instead of raising.

.assert_passed() → None

Run the test case and assert that all checks passed. Raises AssertionError with formatted failure messages if any check fails.

A conversation you captured in production, replayed against checks without calling the agent again:

from giskard.checks import TestCase, Trace, Interaction, Equals

trace = Trace(
    interactions=[
        Interaction(
            inputs="What is the monthly fee on my current account?",
            outputs={"intent": "fee_enquiry", "fee_eur": 3.00},
        ),
        Interaction(
            inputs="And to replace a lost card?",
            outputs={"intent": "fee_enquiry", "fee_eur": 0.00},
        ),
    ]
)

test_case = TestCase(
    name="fee_enquiry_replay",
    trace=trace,
    checks=[
        Equals(
            expected_value=3.00,
            target_key="trace.interactions[0].outputs.fee_eur",
        ),
        Equals(
            expected_value=0.00,
            target_key="trace.interactions[1].outputs.fee_eur",
        ),
    ],
)

result = await test_case.run()

Integration with pytest

assert_passed() raises AssertionError with the check messages, so a failing check reads like any other pytest failure:

import pytest

from giskard.checks import FnCheck, Interaction, TestCase, Trace


@pytest.mark.asyncio
async def test_vague_question_gets_a_follow_up():
    trace = Trace(
        interactions=[
            Interaction(
                inputs="Something is wrong with my account.",
                outputs="Sorry to hear that. Is it a payment, a card, or the balance?",
            )
        ]
    )
    test_case = TestCase(
        name="asks_a_follow_up",
        trace=trace,
        checks=[
            FnCheck(
                fn=lambda t: t.last is not None and "?" in t.last.outputs,
                name="asks_a_follow_up",
            ),
        ],
    )
    await test_case.assert_passed()

`TestCaseResult`

Module: giskard.checks.core.result

Result of test case execution with check outcomes.

TestCaseResult

status TestCaseStatus

Overall test case status (PASS/FAIL/ERROR/SKIP).

results list[CheckResult]

Results from all checks.

passed bool

True if all checks passed.

failed bool

True if any check failed.

errored bool

True if any check errored.

skipped bool

True if all checks were skipped.

duration_ms int

Execution time in milliseconds.

.assert_passed() → None

Raise AssertionError with formatted failure messages if any check did not pass. Useful in pytest.

result = await test_case.run()

if result.passed:
    print("All checks passed!")
else:
    for check_result in result.results:
        if check_result.failed:
            print(f"Failed: {check_result.message}")

# Assert (raises if failed)
result.assert_passed()

`TestCaseRunner`

Module: giskard.checks.testing.runner

Low-level runner for executing test cases. Most users should use test_case.run() instead.

.run() → TestCaseResult

Execute a test case’s checks against its trace.

test_case TestCase Required

The test case to execute.

return_exception bool Default: False

Return results on exceptions.

get_runner() → TestCaseRunner

Get the default process-wide singleton runner instance.

`WithSpy`

Module: giskard.checks.testing.spy

Replace a function with a MagicMock while the wrapped interaction spec runs, then record what it was called with. Use it to assert that your agent passed the right arguments to an internal call.

WithSpy

interaction_generator InteractionSpec Required

The interaction spec to spy on.

target str Required

Python import path of the function to patch, exactly as you would pass it to unittest.mock.patch (for example "myapp.db.fetch_orders"). This is not a JSONPath — a trace expression such as "trace.last.outputs" raises.

After each interaction, the mock’s call history is written to Interaction.metadata under the target string, then the mock is reset. The recorded keys are call_count, call_args, call_args_list, and mock_calls.

Assert that the target called a dependency:

from giskard.checks import Scenario, Interact, WithSpy

from myapp.agent import answer

interaction_spec = Interact(
    inputs="Find item 42",
    outputs=answer,
)

spied_spec = WithSpy(
    interaction_generator=interaction_spec,
    target="myapp.db.fetch_item",  # Python import path, same as mock.patch
)

result = await Scenario("dependency_call").add_interaction(spied_spec).run()

# Spy data is keyed by the same import path
spy_data = result.final_trace.last.metadata.get("myapp.db.fetch_item")
print(spy_data["call_count"], spy_data["call_args"])

Usage patterns

Replaying recorded conversations

Replay a recorded transcript:

from giskard.checks import TestCase, Trace, Interaction, FnCheck

recorded = [
    Interaction(inputs="Find item 42", outputs="Item found."),
    Interaction(inputs="Who created it?", outputs="Alex created it."),
]

trace = Trace(interactions=recorded)

test_case = TestCase(
    name="recorded_replay",
    trace=trace,
    checks=[
        FnCheck(
            fn=lambda t: "found" in t.interactions[0].outputs.lower(),
            name="finds_item",
        ),
        FnCheck(
            fn=lambda t: "Alex" in t.interactions[1].outputs,
            name="returns_creator",
        ),
    ],
)

await test_case.assert_passed()

Replay does not call the agent.

Batch testing

Run independent test cases concurrently:

import asyncio

test_cases = [
    TestCase(
        name=f"replay_{i}",
        trace=Trace(interactions=[interaction]),
        checks=[
            FnCheck(
                fn=lambda t: t.last is not None and bool(t.last.outputs),
                name="answered",
            )
        ],
    )
    for i, interaction in enumerate(recorded)
]

results = await asyncio.gather(*[tc.run() for tc in test_cases])

passed = sum(1 for r in results if r.passed)
print(f"Passed: {passed}/{len(results)}")

Parameterized tests

One check, many recorded exchanges:

import pytest

from giskard.checks import Interaction, StringMatching, TestCase, Trace


def target(inputs: str) -> str:
    return f"Received: {inputs}"


test_data = [
    ("Hello", "Received"),
    ("Goodbye", "Received"),
]


@pytest.mark.asyncio
@pytest.mark.parametrize("question,required_keyword", test_data)
async def test_answers_include_keyword(question, required_keyword):
    trace = Trace(
        interactions=[
            Interaction(inputs=question, outputs=target(question))
        ]
    )
    test_case = TestCase(
        name=f"keyword_{required_keyword}",
        trace=trace,
        checks=[
            StringMatching(
                keyword=required_keyword,
                target_key="trace.last.outputs",
                case_sensitive=False,
            )
        ],
    )
    await test_case.assert_passed()