Generators

Input generators for creating dynamic test data and simulating user interactions.

`UserSimulator`

Simulate realistic user behavior in multi-turn conversations using an LLM. Particularly useful for testing conversational AI agents or chatbots with dynamic, goal-oriented interactions.

Module: giskard.checks.generators.user

instructions str Required

Instructions defining the user’s behavior and goals.

max_steps int Default: 10

Maximum number of interaction steps to generate.

generator BaseGenerator | None Default: None

LLM generator for simulating user behavior.

from giskard.checks.generators.user import UserSimulator
from giskard.agents.generators import Generator

user_sim = UserSimulator(
    instructions="""
    You are a customer trying to book a flight.
    - Start by asking about available flights to Paris
    - Ask about prices
    - Request to book if the price is reasonable
    """,
    max_steps=5,
    generator=Generator(model="openai/gpt-5"),
)

Using in scenarios

from giskard.checks import Scenario, FnCheck

test_scenario = (
    Scenario("booking_flow")
    .interact(
        inputs=user_sim,
        outputs=lambda inputs: booking_agent(inputs),
    )
    .check(
        FnCheck(
            fn=lambda trace: "booked" in trace.last.outputs.lower(),
            name="completed",
        )
    )
)

result = await test_scenario.run()

Goal-oriented testing

result = await test_scenario.run()

output = result.final_trace.last.metadata.get("simulator_output")
if isinstance(output, UserSimulatorOutput):
    print(f"Goal reached: {output.goal_reached}")
    print(f"Message: {output.message}")

`UserSimulatorOutput`

Module: giskard.checks.generators.user

Output format for user simulator results.

goal_reached bool

Whether the simulator achieved its goal.

message str | None

Optional message about the simulation outcome.

`InputGenerator`

Abstract base class for creating custom input generators. Extend it to create generators for dynamic test data.

Module: giskard.checks.core.input_generator

.generate() → Any | None

Generate the next input value. Return None when no more inputs are available.

trace Trace Required

The current trace for context-aware generation.

Sequential generator

from giskard.checks.core.input_generator import InputGenerator


class SequentialInputGenerator(InputGenerator):
    def __init__(self, inputs_list: list[str]):
        self.inputs_list = inputs_list
        self.index = 0

    async def generate(self, trace):
        if self.index >= len(self.inputs_list):
            return None
        input_value = self.inputs_list[self.index]
        self.index += 1
        return input_value


gen = SequentialInputGenerator(["Hello", "How are you?", "Goodbye"])

scenario = Scenario("sequential").interact(
    inputs=gen, outputs=lambda inputs: chatbot(inputs)
)

Context-aware generator

class ContextAwareGenerator(InputGenerator):
    def __init__(self, strategy: str = "follow_up"):
        self.strategy = strategy

    async def generate(self, trace):
        if not trace.last:
            return "Hello, I need help"

        last_output = trace.last.outputs
        if self.strategy == "follow_up":
            if "question" in last_output.lower():
                return "Yes, please tell me more"
            elif "help" in last_output.lower():
                return "I need assistance with my account"
        return "Thank you"

Usage patterns

A/B testing with different personas

from giskard.checks.generators.user import UserSimulator

personas = [
    {
        "name": "impatient",
        "instructions": "You are impatient and want quick answers.",
    },
    {
        "name": "detailed",
        "instructions": "You like detailed explanations. Ask follow-ups.",
    },
    {
        "name": "confused",
        "instructions": "You are confused and need things explained simply.",
    },
]

results = {}
for persona in personas:
    sim = UserSimulator(instructions=persona["instructions"], max_steps=5)
    scenario = Scenario(persona["name"]).interact(
        inputs=sim, outputs=lambda inputs: agent(inputs)
    )
    results[persona["name"]] = await scenario.run()

for name, result in results.items():
    print(f"{name}: {result.status}")