Generators

Input generators for creating dynamic test data and simulating user interactions.

`UserSimulator`

Simulate realistic user behavior in multi-turn conversations using an LLM. Particularly useful for testing conversational AI agents or chatbots with dynamic, goal-oriented interactions.

Module: giskard.checks.generators.user

persona str Required

Predefined persona name (e.g., "frustrated_customer") or a custom persona description.

max_steps int Default: 3

Maximum number of conversation turns to generate.

context str | None Default: None

Optional context to customize the persona’s behavior.

generator BaseGenerator Default: get_default_generator()

LLM generator used to simulate the user’s messages. Defaults to the framework’s default generator.

from giskard.checks.generators.user import UserSimulator
from giskard.agents.generators import Generator

user_sim = UserSimulator(
    persona="""
    You are a customer trying to book a flight.
    - Start by asking about available flights to Paris
    - Ask about prices
    - Request to book if the price is reasonable
    """,
    max_steps=3,
    generator=Generator(model="openai/gpt-5"),
)

Using in scenarios

from giskard.checks import Scenario, FnCheck

test_scenario = (
    Scenario("booking_flow")
    .interact(
        inputs=user_sim,
        outputs=lambda inputs: booking_agent(inputs),
    )
    .check(
        FnCheck(
            fn=lambda trace: "booked" in trace.last.outputs.lower(),
            name="completed",
        )
    )
)

result = await test_scenario.run()

Goal-oriented testing

result = await test_scenario.run()

output = result.final_trace.last.metadata.get("simulator_output")
if isinstance(output, UserSimulatorOutput):
    print(f"Goal reached: {output.goal_reached}")
    print(f"Message: {output.message}")

`UserSimulatorOutput`

Module: giskard.checks.generators.user

Output format for user simulator results.

UserSimulatorOutput

goal_reached bool

Whether the simulator achieved its goal.

message str | None

Optional message about the simulation outcome.

`InputGenerator`

Abstract base class for creating custom input generators. Extend it to create generators for dynamic test data.

Module: giskard.checks.core.input_generator

.__call__() → AsyncGenerator[InputType, TraceType]

Yield input values one at a time. Receives the current trace via send() between yields, so the generator can adapt to prior interactions.

trace TraceType Required

The initial trace passed in when the generator is first called.

Sequential generator

from collections.abc import AsyncGenerator

from giskard.checks.core import Trace
from giskard.checks.core.input_generator import InputGenerator


class SequentialInputGenerator(InputGenerator[str, Trace]):
    inputs_list: list[str]

    async def __call__(self, trace: Trace) -> AsyncGenerator[str, Trace]:
        for value in self.inputs_list:
            trace = yield value


gen = SequentialInputGenerator(inputs_list=["Hello", "How are you?", "Goodbye"])

scenario = Scenario("sequential").interact(
    inputs=gen, outputs=lambda inputs: chatbot(inputs)
)

Context-aware generator

from collections.abc import AsyncGenerator

from giskard.checks.core import Trace
from giskard.checks.core.input_generator import InputGenerator


class ContextAwareGenerator(InputGenerator[str, Trace]):
    strategy: str = "follow_up"
    max_steps: int = 5

    async def __call__(self, trace: Trace) -> AsyncGenerator[str, Trace]:
        # First message — no prior interactions yet.
        trace = yield "Hello, I need help"

        for _ in range(self.max_steps - 1):
            if not trace.last:
                return

            last_output = trace.last.outputs
            if self.strategy == "follow_up":
                if "question" in last_output.lower():
                    trace = yield "Yes, please tell me more"
                    continue
                if "help" in last_output.lower():
                    trace = yield "I need assistance with my account"
                    continue
            trace = yield "Thank you"

Usage patterns

A/B testing with different personas

from giskard.checks.generators.user import UserSimulator

personas = [
    {
        "name": "impatient",
        "persona": "You are impatient and want quick answers.",
    },
    {
        "name": "detailed",
        "persona": "You like detailed explanations. Ask follow-ups.",
    },
    {
        "name": "confused",
        "persona": "You are confused and need things explained simply.",
    },
]

results = {}
for persona in personas:
    sim = UserSimulator(persona=persona["persona"], max_steps=3)
    scenario = Scenario(persona["name"]).interact(
        inputs=sim, outputs=lambda inputs: agent(inputs)
    )
    results[persona["name"]] = await scenario.run()

for name, result in results.items():
    print(f"{name}: {result.status}")